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1  Executive  Summary 

The  Logistics  Readiness  Program  supports  the  Air  Force  Research  Laboratory,  Human  Effectiveness 
Directorate,  Warfighter  Readiness  Research  Division,  Logistics  Readiness  Branch  (AFRL/HEAL) 
through  the  conduct  of  logistics  research  studies  that  leverage  the  expertise  of  the  academic  community. 
This  research  is  the  product  of  a  collaborative  project  conducted  by  personnel  at  AFRL/HEAL  and  the 
University  of  Arkansas.  The  AFRL/HEAL  identified  a  need  to  have  a  strategically  aligned  performance 
measurement  system  for  flightline  maintenance  (MX)  activities.  This  system  must  account  for  the  entire 
flightline  MX  process  in  order  to  improve  the  performance  of  aircraft  scheduling  and  achievement  of 
mission  objectives.  The  primary  project  activities  were: 

♦  Identification  of  a  strategically  aligned  performance  measurement  system 

♦  Research  into  that  system’s  development  and  implementation  process 

♦  Investigation  of  current  flightline  MX  processes 

♦  Production  of  associated  development  guidelines 

♦  Validation  of  these  guidelines  through  a  case  application 

♦  Investigation  of  software  implementations 

The  review  of  the  performance  measurement  literature  resulted  in  the  selection  of  the  Kaplan  and  Norton 
Balanced  Scorecard  (BSC)  as  the  appropriate  measurement  system  for  the  flightline  MX  process. 
Production  of  the  BSC  development  guidelines  required  in-depth  analysis  of  the  traditional  BSC 
development  process  and  the  current  flightline  MX  process  through  documentation  review  and  site  visits. 
The  guidelines  prescribe  the  process  from  selection  of  both  a  target  organization  and  development  team  to 
the  classification  of  measures  and  families  to  the  production  of  a  viable  BSC. 

The  BSC  development  guidelines  were  followed  to  construct  a  preliminary  flightline  MX  BSC.  This  was 
done  to  validate  the  development  guidelines  to  ensure  practical  implementation.  The  resulting  BSC  was 
validated  through  administration  of  a  questionnaire  to  United  States  Air  Force  (USAF)  logistics 
personnel. 

It  was  determined  that  a  review  of  existing  BSC  software  packages  would  benefit  eventual 
implementation  of  the  BSC.  A  review  of  the  three  most  prominent  BSC  software  packages  was  conducted 
based  upon  industry  standards  for  such  packages. 
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The  scope  of  project  is  the  flightline  MX  process.  The  flightline  MX  process  is  the  inspection  and  service 
process  that  takes  place  from  the  time  an  aircraft  lands  through  all  the  activities  necessary  to  recover  the 
aircraft  and  prepare  it  to  successfully  complete  another  mission.  Thorough  knowledge  of  this  process  was 
required  to  determine  the  relevant  performance  measures,  understand  why  the  measures  behave  as  they 
do,  and  indicate  reasons  why  these  measures  fall  outside  acceptable  parameters.  Maintenance  leaders  are 
primarily  concerned  with  knowing  how  well  the  unit  is  meeting  mission  requirements,  improving 
equipment  performance,  identifying  support  problems,  and  projecting  current  trends  (AFLMA,  2001). 
Maintenance  performance  is  generally  assessed  using  standards,  goals,  and  maintenance  plans. 

Kaplan  and  Norton  (1996)  introduced  the  BSC  in  the  early  1990s.  The  BSC  was  introduced  in  an  attempt 
to  reconcile  problems  in  traditional  management  strategies  that  overemphasize  financial  measures  at  the 
expense  of  progress  and  growth.  This  performance  management  system  allows  organizations  to  clarify 
their  strategy  and  assure  that  every  aspect  of  operations  is  directed  toward  the  success  of  these  goals 
(Balanced  Scorecard  Basics,  2003).  When  considering  important  measures  all  at  once,  as  suggested  by  the 
BSC,  management  can  detect  whether  one  area  is  improving  at  the  expense  of  another  area  (Kaplan  and 
Norton,  1996). 

The  BSC  development  process  is  one  that  requires  thorough  knowledge  about  internal  operating 
procedures  and  a  comprehensive  understanding  of  the  system  being  studied.  It  was  determined  that 
detailed  guidelines  for  the  BSC  methodology  would  be  valuable  to  facilitate  the  development  of  BSCs  for 
flightline  MX  personnel.  These  guidelines  assure  that  every  step  is  followed  by  dividing  the  BSC 
development  process  into  three  stages,  Groundwork,  Design,  and  Finalization,  each  comprised  of  multiple 
steps.  The  stages  and  steps  of  the  BSC  Development  Guide  are  listed  below: 

♦  Groundwork  Stage 

■  Team  Selection 

■  Strategic  Framework 

•  Mission  Statement 

•  Core  Values 

•  Vision  Statement 

*  Data  Collection 

•  Process  Data 

•  Strategic  Data 

•  Reference  Materials 
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♦  Design  Stage 

*  Goal  Development 

■  Objectives  Identification 

■  Perspectives  Determination 

■  Performance  Measure  Identification 

■  Measure  to  Family  Assignment 

♦  Finalization  Stage 

■  Measure  Assignment  to  the  BSC 

■  Correlation  Determination 

■  Measure  Finalization 

■  Ownership  Assignment 

■  Scorecard  Cascade 

■  Review  and  Revise 

A  case  study  is  presented  that  describes  the  process  undertaken  by  our  team  to  validate  and  exemplify  the 
BSC  Development  Guide  through  the  development  of  a  preliminary  BSC  for  flightline  MX  activities 
within  an  Aircraft  Maintenance  Unit  (AMU).  The  preliminary  BSC  consists  of  the  following  four 
perspectives  and  measures: 

♦  Mission  Perspective 

■  Maintenance  hours  per  flying  hour 

■  Mission-Capable  (MC)  rate 

■  Partially  Mission-Capable  Maintenance  (PMCM) 

■  Sorties  flown 

■  Totally  Not  Mission-Capable  Maintenance  (TNMCM) 

♦  Influencing  Factors  Perspective 

■  Cannibalization  (CANN)  rate 

■  Mission-Impaired  Capability  Awaiting  Parts  (MICAP)  fill  rates 

■  Maintenance  Scheduling  Effectiveness 

■  Totally  Not  Mission-Capable  Supply  (TNMCS) 
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♦  Management  Perspective 

■  Adherence  to  Operations  (OPS)/Maintenance  Squadron  (MXS)  schedule 

■  Deferred  Discrepancies  (DD)  rate  Awaiting  Maintenance  (AWM) 

■  Total  maintenance  deviations 

■  4-hour  fix  rate 

■  8-hour  fix  rate 

■  12-hour  fix  rate 

■  Days  in  Phase/Isochronal  Inspection  (ISO) 

♦  Internal  Enhancement  Perspective 
*  Cannot  Duplicate  (CND)  rate 

■  Repeat-Recur  (RR)  rate 

■  Special  Experience  Identifiers 

■  Total  abort  rate 

■  Training  schedule  adherence 

■  Upgrade  Training 

■  Unit  average  technical  skill  level 

An  anonymous  questionnaire  was  developed  to  elicit  the  expertise  of  logistics  personnel  in  ranking  the 
criticality  of  the  measures  on  the  preliminary  BSC.  The  questionnaire  was  completed  by  attendees  of  the 
2003  Logistics  Officer  Association  (LOA)  National  Conference.  Twenty-six  viable  questionnaires  were 
collected  and  analyzed.  A  count  of  the  number  of  times  each  perspective  and  each  measure  within  each 
perspective  was  assigned  a  particular  ranking  was  computed.  The  corresponding  percentage  represented 
the  number  of  times  each  perspective  or  measure  was  given  that  ranking  out  of  the  total  number  of 
questionnaires. 

The  results  indicate  that  the  mission  perspective  is  the  most  critical  perspective  with  the  management 
perspective  ranked  as  the  next  most  critical.  The  respondents  indicate  that  the  least  critical  perspective  is 
internal  enhancement  perspective.  Within  the  mission  perspective,  MC  rate  and  PMCM  are  the  most  and 
least  critical  measures  respectively.  The  MICAP  fill  rate  is  ranked  as  the  most  critical  measure  within  the 
influencing  factors  perspective.  Within  the  management  perspective,  the  adherence  to  OPS/MXS 
schedule  is  the  most  critical  measure,  and  the  least  critical  measures  are  DD  rate  AWM  and  4-hour  fix 
rate.  The  CND  rate,  RR  rate,  upgrade  training,  and  unit  average  technical  skill  level  are  the  most  critical 
measures  within  the  internal  enhancement  perspective. 
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In  addition  to  the  comprehensive  results  of  the  respondents  as  a  whole,  the  results  were  categorized  by 
respondent  job  function  level,  specifically  group-level  maintenance  supervision/staff  and  squadron-level 
maintenance  supervision/staff.  The  analysis  was  repeated  for  each  group  individually,  allowing  for 
observation  of  how  the  two  levels  of  job  function  may  differ  in  their  views  of  the  criticality  of  the 
perspectives  or  measures. 

Three  BSC  software  packages,  ActiveStrategy  Enterprise™,  SPImpact,  and  pbviews,  were  reviewed  to 
evaluate  their  adherence  to  relevant  industry  standards.  Relevant  features  range  from  network 
compatibility  to  user  ease  and  friendliness.  It  was  observed  that  pbviews  contains  all  the  preferred 
features. 
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2  Introduction 


2.1  Project  Description 

The  AFRL/HEAL  identified  a  need  to  have  a  strategically  aligned  performance  measurement  system  for 
flightline  MX  activities.  This  performance  measurement  system  must  account  for  the  entire  flightline  MX 
process  in  order  to  improve  the  performance  of  aircraft  scheduling  and  achievement  of  mission 
objectives.  It  is  known  that  successful  achievement  of  mission  objectives  is  the  result  of  a  coordinated 
effort  between  multiple  organizations  and  a  variety  of  processes.  These  coordinated  efforts  require 
proficient  levels  of  preventive  maintenance,  degree  of  training  and  experience,  quality  and  timeliness  of 
suppliers,  and  many  other  factors.  Focusing  on  a  single  component  of  the  flightline  MX  process  to  the 
exclusion  of  other  components  can  cause  short-term  benefits  but  long-term  performance  degradation  and 
failed  objectives. 

The  primary  project  activities  are: 

♦  Identification  of  a  strategically  aligned  performance  measurement  system 

♦  Research  into  that  system’s  development  and  implementation  process 

♦  Investigation  of  current  flightline  MX  processes 

♦  Production  of  associated  development  guidelines 

♦  Validation  of  these  guidelines  through  a  case  application 

♦  Investigation  of  software  implementations 

The  review  of  the  performance  measurement  literature  resulted  in  the  selection  of  the  Kaplan  and  Norton 
BSC  as  the  appropriate  measurement  system  for  the  flightline  MX  process.  Further  review  of  the  BSC 
literature  included  a  sampling  of  the  major  works  in  this  area.  This  review  included  the  history  of  the  BSC 
and  focused  on  the  various  components  of  the  traditional  corporate  BSC  and  the  possible  adaptations  to  a 
defense  organization. 

Production  of  the  BSC  development  guidelines  required  in-depth  analysis  of  the  traditional  BSC 
development  process  and  the  current  flightline  MX  processes  through  documentation  review  and  site 
visits.  The  guidelines  prescribe  the  process  from  selection  of  both  a  target  organization  and  development 
team  to  the  classification  of  measures  and  families  to  the  production  of  a  viable  BSC.  The  methodology 
contains  a  review  and  revision  process  to  ensure  that  the  resulting  BSC  stays  current. 
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To  the  extent  possible,  the  BSC  development  guidelines  were  used  to  develop  a  candidate  flightline  MX 
BSC.  This  was  done  to  validate  the  development  guidelines  to  ensure  practical  implementation.  The 
resulting  BSC  was  validated  through  administration  of  a  questionnaire  to  USAF  logistics  personnel. 

While  strongly  based  upon  feedback  from  USAF  personnel,  this  BSC  was  predominately  externally 
developed.  To  be  truly  strategically  aligned,  a  BSC  must  be  developed  by  the  target  organization.  The 
BSC  is  a  system  that  involves  a  great  deal  of  direct  knowledge  about  the  system  being  monitored.  There 
is  extremely  valuable  process  information  to  be  gained  throughout  the  BSC  development  process. 

It  was  determined  that  a  review  of  existing  BSC  software  packages  would  benefit  eventual 
implementation  of  the  BSC.  A  review  of  the  three  most  prominent  BSC  software  packages  was  conducted 
based  upon  the  existing  standards  for  such  packages. 

Three  BSC  software  packages,  ActiveStrategy  Enterprise™,  SPImpact,  and  pbviews,  were  reviewed  to 
evaluate  their  adherence  to  relevant  industry  standards.  Relevant  features  range  from  network 
compatibility  to  user  ease  and  friendliness.  It  was  observed  that  pbviews  contains  all  of  the  preferred 
features. 

2.2  Objectives 

The  project’s  overall  goal  is  to  formulate  BSC  development  guidelines  that  will  facilitate  strategically 
aligned  performance  measurement  through  the  identification  of  mission-critical  performance  measures 
that  seek  to  improve  the  performance  of  aircraft  scheduling  and  achievement  of  mission  objectives. 

The  project  objectives  are  to: 

♦  Identify  a  strategically  aligned  performance  measurement  system,  i.e.  the  BSC,  through  an  in-depth 
review  of  the  relevant  literature 

♦  Investigate  and  develop  realistic  development  guidelines  through  review  of  BSC  literature,  related 
USAF  documentation  and  site  visits 

♦  Validate  the  BSC  development  guidelines  through  a  case  application  and  practitioner  assessment  of 
the  resulting  BSC 

♦  Investigate  BSC  software  packages  to  help  facilitate  successful  implementation 
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3  Background 

3. 1  Flightline  Maintenance  Process 

The  project  scope  is  the  flightline  MX  process.  The  flightline  MX  process  is  the  inspection  and  service 
process  that  takes  place  from  the  time  an  aircraft  lands  through  all  the  activities  necessary  to  recover  the 
aircraft  and  prepare  it  to  successfully  complete  another  mission.  The  cyclical  process  is  depicted  in  Figure 
3.1  (Flightline  Orientation  Briefing,  2003).  While  each  stage  of  the  process  is  described  briefly  in  this 
section,  the  complete  related  documentation  from  the  orientation  briefing  is  located  in  Appendix  1. 
Thorough  knowledge  of  this  process  is  required  to  determine  the  relevant  performance  measures, 
understand  why  the  measures  behave  as  they  do,  and  indicate  reasons  why  these  measures  fall  outside 
acceptable  parameters. 


Figure  3.1:  Flightline  Maintenance  Process 
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3.1.1  Aircraft  Landing 

The  MX  process  begins  when  an  aircraft  returns  from  a  mission.  As  the  aircraft  lands,  the  aircrew  relays 
system  discrepancies  to  the  flightline.  This  allows  the  flightline  to  be  prepared  for  the  discrepancies  and 
make  repairs  faster. 

3.1.2  Parking  and  Recovery 

The  parking-and-recovery  stage  neutralizes  the  threat  of  munitions  and  fuel  explosions  through  grounding 
and  stabilizes  landing  gear.  The  processes  in  this  stage  should  be  strictly  monitored  for  safety  standards 
adherence.  Performance  measures  used  for  this  process  should  reflect  an  emphasis  on  safety. 

3.1.3  Aircraft  Servicing 

Aircraft  servicing  actions  include  checking  system  fluids  and  refueling.  This  is  another  stage  where  safety 
is  paramount.  Essential  measures  include  training  measures  to  ensure  that  training  is  up-to-date  and  the 
most  recent  practices  are  adhered  to. 

3.1.4  Aircrew  Debriefing 

Aircrew  maintenance  debriefing  is  the  communication  between  the  aircrew  and  the  MX  crew.  The 
debriefing  has  a  large  impact  on  the  quality  of  repairs  completed  by  the  MX  crew.  A  complete  debriefing 
is  an  important  factor  for  quality  performance  measures  such  as  repeat  recurrences.  This  process  is  aided 
by  the  use  of  different  codes  for  landing  status,  system  capability,  and  deviations. 

3.1.5  Unscheduled  Maintenance/Repair 

The  success  of  the  unscheduled  MX  and  scheduled  MX  operations  indicate  the  health  of  the  supply  chain 
through  the  length  of  time  to  receive  parts  and  the  number  of  backorders  on  those  parts. 

3.1.6  Preventive  Maintenance 

Preventive  MX  operations  can  be  good  indicators  of  the  ability  of  a  MX  unit  to  manage  its  schedule 
efficiently.  Operations  such  as  Time  Compliance  Technical  Orders  (TCTO),  Time  Change  Item  (TCI) 
replacements,  and  system  calibrations  further  reduce  wear  on  aircraft  and  increase  a  MX  unit’s  ability  to 
keep  planes  in  the  air. 
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3.1.7  Aircraft  Scheduled  for  Next  Mission 

Aircraft  scheduling  for  the  next  mission  and  aircraft  mission  preparation  are  essential  for  successful 
mission  completion.  In  this  stage,  aircraft  are  scheduled  and  prepared  for  various  missions.  Success  in 
these  stages  has  a  direct  impact  on  the  number  of  maintenance-chargeable  deviations  incurred. 

3.1.8  Pre-launch  Inspection 

Pre  launch  inspection  involves  detailed  MX  and  aircrew  inspections  including  visual  examination  of  the 
aircraft  and  operationally  checking  certain  systems  and  components. 

3.1.9  Aircraft  Launch 

The  aircraft  launch  requires  the  aircrew  to  start  the  engines,  power  up  systems,  and  make  final 
adjustments  in  preparation  for  launch. 

3.1.10  Post-launch  Check-up 

The  last  step  in  the  process  involves  cleaning  up  the  parking  location. 

3.2  Maintenance  Performance  Measurement 

Maintenance  leaders  are  primarily  concerned  with  knowing  how  well  the  unit  is  meeting  mission 
requirements,  improving  equipment  performance,  identifying  support  problems,  and  projecting  current 
trends  (AFLMA,  2001).  It  is  pertinent  that  MX  leaders  review  sortie  production  and  MX  performance 
constantly  and  are  knowledgeable  about  predictive  MX  indicators.  Maintenance  performance  is  generally 
assessed  using  standards,  goals  and  maintenance  plans.  The  following  performance  measurement 
questions  were  developed  by  the  AFLMA  (2001): 

♦  Are  operational  requirements  based  upon  realistic  availability  of  equipment? 

♦  What  are  the  causes  of  flying  schedule  deviations? 

♦  Are  particular  aircraft,  equipment,  systems,  or  subsystems  contributing  to  a  disproportionate  share  of 
deviations? 

♦  Does  specific  equipment  fail  to  perform  as  scheduled  or  require  more  or  less  MX  than  normal? 

♦  Is  there  enough  staff  to  meet  mission  needs? 

♦  Do  higher  rates  of  repeat/recur  discrepancies  indicate  training  or  experience  shortfalls? 

♦  Is  there  sufficient  time  to  schedule  and  work  MX  problems? 

♦  What  is  the  behavior  of  MX  trends? 
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There  are  two  prominent  types  of  performance  indicators  associated  with  maintenance:  leading  and 
lagging.  Leading  indicators  directly  impact  MX  capability  to  provide  resources  to  execute  the  mission  by 
measuring  performance  before  a  problem  arises.  Lagging  indicators  provide  information  after  the  problem 
has  occurred  and  can  indicate  firmly  established  trends. 

3.2.1  Leading  Indicators 

Maintenance  Scheduling  Effectiveness  (MSE)  Rate  -  The  number  of  maintenance  actions  started  as 
scheduled  per  total  number  of  MX  actions  scheduled.  MSE  rate  measures  maintenance’s  ability  to  plan 
and  complete  inspections  and  scheduled  MX. 

DD  Rate  -  DD  rate  depicts  how  well  the  unit  is  keeping  up  with  required  minor  repairs.  Deferred 
discrepancies  are  minor  MX  actions  that  may  be  deferred  until  a  more  opportune  time. 

3.2.2  Lagging  Indicators 

Fully  Mission-Capable  (FMC)  Rate  -  A  low  FMC  rate  may  indicate  a  parts  problem.  This  measure 
should  be  compared  with  the  monthly  MC  rate.  Significant  differences  may  indicate  that  aircraft  are 
flying  partially  inoperable. 

TNMCS  Rate  -  This  rate  is  based  upon  the  number  of  airframes  out  for  parts;  therefore,  spare  part 
availability  is  critical  for  this  measure.  Maintenance  can  reduce  this  rate  by  limiting  the  number  of 
CANNs. 

3.3  Balanced  Scorecard 

Kaplan  and  Norton  (1996)  introduced  the  Balanced  Scorecard  (BSC)  in  the  early  1990s.  The  BSC  was 
introduced  in  an  attempt  to  reconcile  problems  in  traditional  management  strategies.  Traditional 
management  strategies  overemphasized  financial  measures  at  the  expense  of  progress  and  growth.  This 
overemphasis  brought  about  short-term  gains  to  the  detriment  of  long-term  success.  The  BSC  is  a 
performance  management  system  that  allows  organizations  to  clarify  their  strategy  and  assure  that  every 
aspect  of  operations  is  directed  toward  the  success  of  these  goals  (Balanced  Scorecard  Basics,  2003). 

When  considering  important  measures  all  at  once,  as  suggested  by  the  BSC,  management  can  detect 
whether  one  area  is  improving  at  the  expense  of  another  area  (Kaplan  and  Norton,  1996).  Kaplan  and 
Norton  suggest  the  following  analogy  to  better  explain  the  purpose  of  the  BSC.  In  a  cockpit,  a  large  and 
complex  amount  of  data  is  displayed  very  quickly  and  simply  through  the  use  of  cockpit  displays.  These 
display  fuel  level,  airspeed,  altitude,  bearing,  and  destination.  Focusing  on  just  one  instrument  can  be 
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fatal,  just  as  focusing  on  one  aspect  of  performance  can  be  fatal  to  operational  success.  A  BSC  is  designed 
to  display  all  pertinent  performance  information  simultaneously. 


Kaplan  and  Norton  state  that,  in  addition  to  the  traditional  financial  performance  measures,  the  BSC 
incorporates  non-fmancial  measures  that  enable  value  creation  for  the  organization  (Kaplan  and  Norton, 
1996).  These  measures  have  focused  on  managing  intangible  assets  such  as  customer  relationships,  skills 
and  knowledge  of  the  workforce,  and  the  technology  that  supports  the  workers  (Kaplan  and  Norton, 
2001).  A  graphical  representation  of  the  BSC  is  depicted  in  Figure  3.2  (Kaplan  and  Norton,  1996). 


Figure  3.2:  BSC  Framework 


The  original  four  perspectives  of  the  balanced  scorecard  (financial,  customer,  internal  processes,  and 
learning  and  growth)  are  focused  on  the  mission  or  strategic  plan  of  an  organization.  In  order  for  an 
organization  to  achieve  its  goals,  it  is  important  that  any  performance  measurement  system  be  aligned 
with  and  contributing  to  the  overall  mission  of  the  organization. 
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3.3.1  Financial  Perspective 

The  financial  perspective  is  the  objective  perspective  and  represents  factors  that  contribute  to  successful 
company  cash  flow  and  market  share.  Although  financial  measures  may  provide  management  with 
numeric  data  concerning  the  company’s  performance,  these  are  lagging  indicators  of  past  events.  Crandall 
(2002)  states  that  traditional  performance  measurement  systems  rely  too  heavily  upon  financial  measures 
and  neglect  issues  such  as  quality  and  customer  service.  Tracking  the  performance  of  past  events  through 
financial  performance  measures  is  important;  however,  non-financial  measures  that  help  create  value 
should  also  be  considered. 

3.3.2  Customer  Perspective 

The  customer  perspective  is  more  closely  related  to  the  company’s  mission  statement  than  the  widely 
implemented  financial  measures.  A  large  number  of  companies  today  have  a  mission  statement  that 
focuses  on  customers  (Kaplan  and  Norton,  1996).  This  perspective  incorporates  customer  issues  into  the 
company’s  performance  measurement  system.  Taking  customer  concerns  into  account  is  a  critical  aspect 
of  operating  a  business  and  should  therefore  be  included  in  the  company’s  performance  measurement 
system. 

3.3.3  Internal  Business  Perspective 

The  internal  business  perspective  is  concerned  with  measures  such  as  timeliness  of  operations,  actual  vs. 
planned  activities,  and  issues  with  competition.  This  perspective  is  intended  to  measure  what  the  company 
is  doing  internally  to  address  the  needs  and  wants  of  the  customer.  The  internal  measures  associated  with 
the  BSC  are  derived  from  internal  processes  that  have  a  significant  impact  on  customer  satisfaction. 
Kaplan  and  Norton  (2001)  state  that  this  perspective  captures  an  organization’s  activities.  Achievement  of 
operational  excellence  through  the  improvement  of  internal  processes  and  supply  chain  management  are 
activities  that  should  be  captured  with  this  perspective  (Kaplan  and  Norton,  2001). 

3.3.4  Learning  and  Growth  Perspective 

The  learning  and  growth  perspective  is  the  foundation  for  corporate  strategy.  This  is  the  perspective 
where  management  assesses  their  employee  ability,  technological  status,  and  corporate  climate  needed  to 
support  a  strategy  (Kaplan  and  Norton,  2001).  This  perspective  includes  training  employees  and  shaping 
attitudes  to  promote  self-improvement  (Balanced  Scorecard  Basics,  2003).  It  is  important  that  the 
organization  has  long-term  goals  and  a  strategy  in  place  with  targets  to  be  achieved  to  promote  growth. 
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4  BSC  Development  Guide 

The  BSC  development  process  is  one  that  requires  thorough  knowledge  about  internal  operating 
procedures  and  a  comprehensive  understanding  of  the  system  being  studied.  It  was  determined  that 
detailed  guidelines  for  the  BSC  methodology  would  be  valuable  to  facilitate  the  development  of  BSCs  for 
flightline  MX  personnel.  These  guidelines  are  designed  for  use  by  USAF  personnel  with  little  or  no  BSC 
experience  to  develop  scorecards  for  the  flightline  MX  process.  Adherence  to  the  development  guidelines 
is  pertinent  for  developing  a  strategically  aligned  BSC.  The  guidelines  assure  that  every  step  is  followed 
by  dividing  the  BSC  development  process  into  manageable  stages.  Figure  4.1  depicts  the  BSC 
development  process,  which  consists  of  the  three  primary  stages:  Groundwork,  Design  and  Finalization. 
Each  stage  is  comprised  of  multiple  steps,  which  are  detailed  in  the  remainder  of  this  section. 
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Figure  4.1:  BSC  Process  Diagram 


4.1  Groundwork  Stage 

The  groundwork  stage  is  the  foundation  of  a  successful  BSC  project.  By  following  the  steps  in  this  stage, 
all  the  pertinent  and  necessary  information  needed  for  the  scorecard  will  be  available  during  construction. 

A  thorough  groundwork  stage  will  assure  that  key  information  is  not  overlooked.  It  will  also  assure  that 
the  gathered  information  is  organized  in  a  logical  manner. 

4.1.1  Team  Selection 

The  BSC  should  be  undertaken  by  a  team  of  stakeholders,  not  a  single  person.  This  allows  for 
brainstorming  and  group  discussion.  It  also  facilitates  thorough  review  of  report  documents.  The  team 
should  be  comprised  of  personnel  from  all  involved  functions  and  should  represent  the  skill  sets  in  the 
organization.  Inclusion  of  all  stakeholders  will  facilitate  acceptance  and  enthusiasm  for  BSC 
implementation.  The  members  of  the  BSC  team  will  act  as  ambassadors  for  the  BSC,  accelerating  its 
acceptance  and  use  (Niven,  2002).  Each  team  member  should  be  qualified  to  provide  expert  opinion  about 
his  or  her  organizational  function.  An  understanding  of  BSC  concepts  and  its  development  process  is 
imperative  for  all  team  members.  Skill  in  quantitative  analysis  must  also  be  present  among  the  team  as 
this  is  extremely  useful  during  the  finalization  of  measures. 

While  team  size  is  ultimately  up  to  the  team  leader,  a  standard  BSC  team  has  approximately  seven 
members.  In  general,  more  than  seven  members  may  create  difficulties  in  coordinating  group  efforts;  less 
than  seven  may  not  bring  enough  viewpoints  to  the  process.  The  team  leader  is  responsible  for 
coordinating  all  team  meetings  and  corresponding  with  superiors  (Niven,  2002).  Other  members  of  the 
team  may  be  assigned  specific  duties  as  needed  during  the  development  process. 

The  goal  for  team  member  selection  is  to  represent  as  many  levels  of  the  organization  as  possible.  The 
reasoning  behind  this  goal  is  that  a  higher-level  supervisor,  such  as  a  unit  commander,  understands 
overall  strategy  and  desired  outcomes  of  the  organization.  In  addition,  there  are  many  low-level  aspects  of 
operations  that  are  important  to  performance  measurement.  These  are  better  understood  by  the  persons 
who  work  at  this  level  every  day. 

4.1.2  Strategic  Framework 

Organizational  strategy  is  the  guiding  factor  behind  the  BSC.  Organizational  strategy  is  defined  as  a  set  of 
long-term  goals  that,  if  successfully  achieved,  will  revolutionize  the  way  a  unit  operates.  Without 
strategic  alignment  or  the  integration  of  this  organizational  strategy  into  the  BSC,  a  BSC  is  merely  a 
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collection  of  performance  measures.  Strategic  planning  and  alignment  to  a  given  strategy  should  be  the 
top  priority  in  any  BSC  venture.  There  is  a  large  gap  between  having  a  good  strategy  and  effectively 
implementing  it.  The  BSC  provides  a  framework  to  transition  from  deciding  to  have  a  strategy  and 
actually  using  it  (Niven,  2002).  The  next  section  discusses  gathering  a  few  basic  strategic  components. 
Identification  of  an  organization’s  strategy  will  help  determine  the  most  relevant  data  to  collect. 

The  process  of  strategic  alignment  begins  at  the  top  of  the  participating  organization,  regardless  of  its 
scope  or  size.  The  unit  commander  and  relevant  subordinates  must  come  together  to  determine  what  the 
organizational  strategy  is  and  where  opportunities  for  achievement  of  this  strategy  exist.  This  is  a 
complex  process  requiring  time  and  effort.  There  is  often  disparity  between  the  commander  and  other 
members  of  the  organization  on  how  the  organizational  strategy  is  to  be  implemented.  In  many  cases,  a 
documented  strategy  does  not  exist.  In  this  case,  a  sound  organizational  strategy  must  be  developed.  The 
essential  strategic  elements  for  a  successful  BSC  are  mission,  core  values,  and  a  vision  statement  (Niven, 
2002).  If  these  strategic  elements  are  already  in  existence  and  are  approved  by  the  team,  they  can  be 
integrated  into  the  framework.  Otherwise,  these  elements  must  be  developed  as  described  in  Sections 
4. 1.2.1  through  4.1. 2. 3. 

4.1. 2.1  Mission  Statement 

Mission  statements  have  been  adopted  by  almost  every  organization  in  existence.  They  are  used  to 
communicate  fundamental  beliefs  and  identify  target  goals  (Kaplan  and  Norton,  1996).  A  mission 
statement  should  be  motivating  and  inspiring.  An  effective  mission  statement  is  not  something  that 
changes  every  year  but  lasts  for  many  years  as  a  foundation  for  the  organization.  A  mission  statement 
should  be  easily  understood  and  communicated  down  to  the  lowest  level  of  the  organization  (Niven, 
2002).  An  example  mission  statement  is  that  of  the  USAF:  “To  defend  the  United  States  through  control 
and  exploitation  of  air  and  space.” 

4.1. 2.2  Core  Values 

Niven  holds  that  “values  are  the  timeless  principles  that  guide  an  organization”  (Niven,  2002).  These 
principles  are  deeply  held  beliefs  that  exist  within  the  organization  and  are  demonstrated  through  the  day- 
to-day  behaviors  of  all  employees.  These  values  set  the  tone  for  an  organization  by  telling  each  member 
of  the  unit  how  to  accomplish  his  or  her  mission.  For  example,  the  core  values  of  the  USAF  are: 

♦  Integrity  -  Do  the  job  right  the  first  time 

♦  Service  -  Mission  accomplishment  over  personal  gain 

♦  Excellence  -  Put  forth  the  best  possible  effort  all  the  time 
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4.1. 2.3  Vision  Statement 

A  vision  statement  is  a  snapshot  of  the  future.  An  excellent  example  of  a  vision  statement  is  the  USAF’s 
“ Global  Vigilance,  Reach,  and  Power:  Vision  2020”  (Vision  2020,  2000).  It  contains  multiple  long-term 
goals  that  can  take  anywhere  from  several  years  to  a  few  decades  to  achieve.  Many  of  the  long-term  goals 
from  a  vision  statement  can  help  to  define  the  characteristics  of  the  BSC  perspectives.  It  is  important  to 
avoid  vague  catchwords  and  phrases.  The  use  of  very  technical  words  is  also  discouraged  since  all 
stakeholders  may  not  be  familiar  with  such  language.  It  should  be  clear  to  all  stakeholders,  not  just  the 
upper  command,  where  the  organization  is  going  and  exactly  how  they  plan  to  get  there  (Brown,  1996). 

4.1.3  Data  Collection 

Data  collection  is  an  essential  step  in  the  BSC  development  process.  The  following  subsections  describe 
the  key  types  of  data  and  data  collection  activities.  The  development  of  a  BSC  generates  large  amounts  of 
data.  This  data  must  be  organized  and  stored  in  a  logical  manner  to  prevent  contamination  or  loss  of 
pertinent  information.  One  team  member  should  be  delegated  the  responsibility  of  organizing  the  data  and 
keeping  it  up-to-date  by  maintaining  the  latest  version  of  all  relevant  documentation.  Master  copies  of  all 
reference  materials  should  be  accessible  in  a  secure,  central  location.  This  protects  the  data  and  provides 
ready  access  to  team  members. 

4.1 .3.1  Process  Data 

The  first  step  in  data  collection  is  to  collect  data  on  the  processes  that  are  to  be  monitored  by  the  BSC. 
Details  on  process  data  collection  are  as  follows: 

♦  Determine  exactly  what  each  process  is  and  locate  any  existing  documentation  about  the  process 

♦  Determine  the  chain  of  command  for  the  process 

♦  Document  the  current  process  as  it  occurs  in  the  organization 

♦  Compare  and  contrast  the  existing  and  team-developed  documentation  about  the  current  process. 

4.1. 3.2  Strategic  Data 

The  next  step  is  collecting  strategic  planning  data  from  the  highest-level  command.  This  data  collection 
does  not  exclusively  involve  the  highest-level  commander,  but  all  those  involved  in  strategic  planning  or 
leadership  of  the  unit.  This  information  is  essential  for  the  strategic  planning  portion  of  BSC 
development.  Specifically,  it  will  help  determine  what  measures  should  be  monitored  and  how  they 
should  be  linked  throughout  the  scorecard.  This  data  should  be  gathered  in  personal  interviews  with  the 
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commander  and  others  involved  in  the  strategic  planning  process.  The  following  are  pertinent  questions  to 
have  answered  (Niven,  2002): 

♦  What  is  your  interpretation  of  the  Mission  Statement,  Core  Values,  and  Vision  Statement? 

♦  Who  are  your  customers? 

♦  What  key  strategies  will  help  to  achieve  your  vision? 

♦  How  can  these  strategies  be  achieved? 

♦  What  measures  or  data  do  you  track  to  monitor  success? 

♦  What  targets  do  you  use  for  these  measures? 

♦  What  related  reports  do  you  find  most  useful? 

4.1. 3.3  Reference  Materials 

Reference  materials  are  published  documents  that  contain  information  on  processes  and  their  performance 
measurement.  Reference  materials  also  include  information  pertaining  to  the  BSC  and  its  application. 
These  materials  assist  in  identifying  measures  and  perspectives  for  the  scorecard  and  indicate  the  relative 
importance  of  measures  or  processes.  It  is  essential  to  gather  materials  from  as  many  different  sources  as 
possible.  Following  is  a  sample  of  potential  sources: 

♦  Published  manuals  and  training  guides  -  These  documents  provide  information  on  what 
performance-related  information  the  unit  already  monitors.  Manuals  are  an  excellent  information 
source  for  potential  BSC  measures  and  instruction  on  how  they  should  be  monitored. 

♦  Health  of  Fleet  (HOF)  Reports  -  HOF  reports  are  a  source  for  many  measures  that  are  already 
tracked  and  reported.  Since  these  measures  are  already  in  operation,  it  is  much  easier  to  integrate 
these  measures  into  a  BSC. 

♦  9302  Reports  -  Since  performance  measurement  of  the  measures  on  the  9302  reports  are  required  by 
Air  Combat  Command  (ACC),  these  measures  are  considered  vitally  important. 

♦  Public  Literature  -  Published  research  provides  information  on  the  BSC  and  its  development  and 
implementation. 

♦  Performance  Documents  -  Any  documents  other  than  those  named  above  that  contain  relevant 
performance  measures  should  be  used. 

♦  Expert  interviews  -  Personnel  with  a  high  level  of  experience  in  their  unit  or  career  track  can  often 
suggest  additional  important  measures  other  than  those  currently  being  monitored. 
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4.2  Design  Stage 

The  design  stage  begins  with  structuring  the  strategic  elements  of  the  organization  and  progresses  through 
performance  measure  identification  and  construction  of  the  basic  BSC  framework. 

4.2.1  Goal  Development 

Strategic  planning  informs  everyone  in  an  organization  where  he  or  she  is  going  and  how  to  get  there. 
Effective  development  of  a  strategy  was  outlined  in  Section  4.1.2.  The  conversion  of  this  strategy  into 
goals  is  the  topic  of  this  section.  The  overall  strategy  must  be  broken  down  into  long-term  goals.  These 
are  goals  that  can  revolutionize  the  way  an  organization  operates  by  taking  them  from  their  current  state 
to  envisioned  future  success.  These  goals  should  be  future-focused  with  a  time  frame  ranging  between 
five  to  20  years.  Milestones  should  be  identified  to  divide  the  long-term  goals  into  shorter  time  buckets. 
Milestones  are  subdivisions  of  a  goal  that  are  used  to  check  progress  toward  achievement  of  the  goal. 
These  long-term  goals  can  then  be  transformed  into  performance  measures  later  in  the  BSC  development 
process.  Each  goal  must  have  a  specific  target  that  is  either  numerical  or  descriptive.  These  targets  are  the 
projected  optimal  result  for  each  goal. 

Brown  (1996)  identifies  five  common  problems  associated  with  setting  goals,  as  listed  below.  All 
pertinent  goals  should  be  reviewed  to  verify  that  they  do  not  contain  any  of  these  flaws. 

♦  Goals  that  are  really  projects,  activities,  or  strategies  -  The  best  way  to  avoid  this  is  to  ensure  that 
each  goal  has  at  least  one  measure  in  the  scorecard. 

♦  Goals  that  are  solely  based  upon  past  performance  -  Many  organizations  simply  add  five  or  ten 
percent  to  last  year’s  goal  without  justification. 

♦  Arbitrary  stretch  goals  -  Developing  a  goal  without  good  reason  or  randomly  selecting  a  competitor’s 
goal  should  be  avoided. 

♦  Inconsistent  short-term  and  long-term  goals  -  All  short-term  goals  should  be  components  of  some 
long-term  goal. 

♦  Inconsistencies  in  goals  at  different  levels  of  the  organization  -  Every  goal  should  cascade  down  from 
a  higher  goal. 

4.2.2  Objectives  Identification 

Identifying  objectives  is  a  translation  of  strategy  and  long-term  goals  into  specific  timelines  and  events. 
Each  long-term  goal  has  a  realistic  target,  and  milestones  have  been  identified  as  a  portion  of  that  target. 
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These  goals  and  targets  will  be  placed  as  measures  in  the  objective  perspective  to  show  success  in 
strategic  objectives. 

4.2.3  Perspectives  Determination 

As  previously  discussed  in  Section  3.3,  the  original  Kaplan  and  Norton  (1996)  BSC  suggested  four 
perspectives: 

♦  Financial  Perspective 

♦  Customer  Perspective 

♦  Internal  Business  Process  Perspective 

♦  Learning  and  Growth  Perspective 

Kaplan  and  Norton  recognize  these  four  perspectives  “should  be  considered  a  template,  not  a  straight 
jacket.”  Their  perspectives  are  intended  to  portray  the  essential  elements  that  can  lead  to  success  in  a 
typical  organization  (Kaplan  and  Norton,  1996).  Although  four  is  standard,  there  is  no  set  rule  for 
determining  the  number  of  perspectives  in  a  scorecard.  While  fewer  than  four  is  uncommon,  there  are 
many  instances  of  more  than  four.  If  there  are  more  than  four  key  elements  that  give  a  competitive  edge 
or  portray  key  competencies,  these  should  all  be  included  as  perspectives;  however,  care  should  be  taken 
when  adding  perspectives,  because  too  many  perspectives  can  lead  to  scorecards  with  large  numbers  of 
stand-alone  perspectives  that  are  unrelated  to  each  other.  Niven  (2002)  suggests  that  the  true  test  of 
perspectives  is  whether  they  can  be  intertwined  to  tell  a  coherent  story.  As  shown  in  Figure  4.2  below, 
success  in  any  one  perspective  can  be  linked  to  success  in  the  others.  Improvements  in  lower  perspectives 
lead  to  good  results  in  higher  perspectives,  which  then  lead  to  realization  of  the  vision  (Niven,  2002).  This 
linkage  is  further  discussed  in  Section  4.3.2. 
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Vision  and  Strategy 


FINANCIAL  PERSPECTIVE 
How  will  successful 
mission  completion 
appear  to  superiors? 


MANAGEMENT  PERSPECTIVE 
To  accomplish  the  mission 
objectives,  what  must  be  done  for 
customers  or  superiors? 


INTERNAL  ENHANCEMENT 
PERSPECTIVE 
In  what  internal  areas  must 
excellence  by  achieved? 


4.2.4  Performance  Measure  Identification 

At  this  point  in  the  development  process,  there  is  a  success  strategy  mapped  out,  and  objectives  for 
success  have  been  determined.  Using  these  objectives,  the  BSC  perspectives  have  been  identified.  The 
next  step  is  to  determine  possible  measures  for  inclusion  in  the  BSC.  Using  the  materials  gathered  during 
the  groundwork  stage,  all  relevant  performance  measures  should  be  identified  and  listed.  Each  document 
should  be  thoroughly  reviewed  and  all  identified  measures  should  be  compiled.  Equations  or  formulae 
used  to  calculate  the  measures  should  be  included  along  with  any  targets  for  each  measure.  This  list 
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serves  as  a  pool  from  which  to  draw  the  key  measures  for  the  performance  objectives.  This  list  is  not  yet 
the  master  list  of  measures;  this  is  the  list  of  measures  that  are  currently  being  tracked.  During  the 
development  of  the  performance  objectives,  it  may  become  apparent  that  additional  measures  that  are  not 
currently  tracked  are  needed  to  track  the  BSC  objectives.  These  new  measures  will  become  more  apparent 
as  the  project  progresses  and  should  be  listed  separately  as  they  are  identified.  Along  with  all  measures 
listed,  it  is  useful  to  have  a  set  of  parameters  such  as:  maximum,  minimum,  optimal,  and  benchmark  (how 
other  similar  organizations  perform).  Data  without  goals  or  comparisons  are  meaningless  (Brown,  1 996). 
For  example,  knowing  that  the  phase  average  for  a  B-52  wing  is  21 5  hours  is  not  useful  information 
unless  it  is  known  that  the  phase  average  should  be  approximately  1 50  hours.  The  comparison  data  is 
what  indicates  the  actual  performance  of  a  measure. 

4.2.5  Measure  to  Family  Assignment 

After  the  list  of  measures  is  compiled,  a  logical  organization  needs  to  take  place.  The  measures  are 
assigned  to  families  based  upon  similar  characteristics  and/or  applications.  An  example  of  a  measure  and 
its  family  is  “sorties  flown”  and  the  “Productivity  Family.”  “Sorties  flown”  measures  productivity  when 
used  in  the  context  of  a  maintenance  unit;  therefore,  it  is  assigned  to  the  family  of  productivity-related 
measures.  Assigning  measures  to  families  is  an  initial  starting  point  to  the  construction  of  the  BSC.  Once 
families  are  created,  they  are  assigned  to  perspectives  in  the  BSC.  It  is  important  to  note  that  each  family 
should  be  included  in  only  one  perspective;  however,  one  perspective  can  be  comprised  of  more  than  one 
family.  This  facilitates  a  smooth  process  for  adding  measures  to  the  scorecard.  If  a  productivity  measure 
is  required,  it  can  be  drawn  from  the  Productivity  Family.  If  a  measure  becomes  too  expensive  or 
cumbersome  to  measure,  it  can  be  replaced  by  another  similar  measure  from  the  same  family. 

4.3  Finalization  Stage 

The  finalization  stage  is  the  continuous  improvement  stage  of  the  development  process.  This  cyclical 
stage  includes  assignment  and  reassignment  of  measures  to  the  BSC  based  upon  their  pertinence  to  the 
ever-changing  activities  and  current  strategies  of  the  organization.  This  stage  continues  over  the  life  cycle 
of  the  BSC. 

4.3.1  Measure  Assignment  to  the  BSC 

An  exhaustive  list  of  currently  tracked  performance  measures  and  candidate  new  measures  now  exist.  The 
next  step  is  to  select  measures  from  this  list  for  inclusion  in  the  BSC.  In  the  measure  selection  process,  it 
is  important  to  note  that  a  single  person  or  scorecard  should  monitor  no  more  than  20  measures  (Brown, 
1996).  If  necessary,  measures  can  be  combined  into  aggregate  measures.  Aggregation  of  measures  is 
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discussed  in  Section  4.3.5.  When  selecting  measures,  the  most  important  factor  is  to  ensure  that  each 
chosen  measure  reflects  the  strategies  developed  earlier  in  the  process. 


Caplice  and  Sheffi  (1994)  provide  eight  evaluation  criteria  that  can  be  used  to  pare  down  the  exhaustive 
list  of  measures  into  the  measures  that  will  eventually  be  included  in  the  BSC.  These  eight  criteria  are 
listed  and  defined  in  Table  4.1.  Trade-offs  exist  between  these  criteria  and  will  need  to  be  evaluated  based 
upon  organizational  priorities. 


Table  4.1:  Eight  Criteria  Measures 


Criterion 

Description 

Validity 

The  measure  accurately  captures  the  events  and  activities  being  measured 
and  controls  for  any  exogenous  factors 

Robustness 

The  measure  is  interpreted  similarly  by  the  users,  is  comparable  across  time, 
location,  and  organizations,  and  is  repeatable. 

Usefulness 

The  measure  is  readily  understandable  by  the  decision  maker  and  provides  a 
guide  for  action  to  be  taken. 

Integration 

The  measure  includes  all  relevant  aspects  of  the  process  and  promotes 
coordination  across  functions  and  divisions. 

Economy 

The  benefits  of  using  the  measure  outweigh  the  costs  of  collection,  analysis, 
and  reporting. 

Compatibility 

The  measure  is  compatible  with  the  existing  information,  material,  cash  flows 
and  systems  in  the  organization. 

Level  of  Detail 

The  measure  provides  a  sufficient  degree  of  granularity  or  aggregation  for  the 
user. 

Behavioral  Soundness 

The  measure  minimizes  incentives  for  counter-productive  acts  or  game 
playing  and  is  presented  in  useful  form. 

Specific  examples  of  criteria  trade-offs  are  discussed  next.  A  measure  that  is  behaviorally  sound  may  be 
very  uneconomic.  That  is,  the  measure  may  prevent  cheating  the  system  very  well,  but  the  cost  of  the 
required  supervision  is  prohibitively  expensive.  Another  possible  trade-off  exists  between  the  criteria 
usefulness  and  level  of  detail.  A  measure  that  is  readily  understandable  may  be  so  watered  down  that  it  is 
a  poor  reflection  of  what  is  actually  going  on  and  therefore  useless.  One  method  for  making  priority  trade¬ 
offs  between  these  criteria  is  the  use  of  a  weighting  system.  Each  criterion  is  assigned  a  priority  weight 
with  the  sum  of  all  the  weights  normalized  to  100  percent.  Each  measure  is  then  assigned  a  weight 
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corresponding  to  its  value/importance  to  each  criterion.  The  weighted  sum  of  each  measure  is  calculated 
and  the  measures  with  the  highest  ratings  are  selected.  This  provides  an  efficient,  quantitative  method  for 
deciding  between  similar  measures;  however,  this  should  not  be  the  only  selection  criterion.  Team 
expertise  and  opinion  on  strategic  prioritization  should  be  employed  when  deciding  between  measures. 

4.3.2  Correlation  Determination 

This  can  be  the  most  challenging  step  in  the  development  of  an  effective  BSC.  A  scorecard  without 
strategic  linkages  is  simply  a  group  of  unrelated  performance  measures.  The  key  is  to  determine  how  the 
strategy  relates  to  each  perspective  in  the  BSC.  Determining  correlations,  for  example,  the  way  in  which 
each  perspective  contributes  to  the  success  of  the  overall  strategy,  begins  with  the  objective  perspective. 
This  perspective  contains  the  strategic  goals  for  an  organization,  and  all  improvements  elsewhere  in  the 
BSC  should  positively  affect  it.  The  correlation  process  works  through  each  perspective,  showing  how 
each  perspective  relates  to  the  objective  perspective.  Next  is  an  example  to  exemplify  this  process. 

The  correlation  process  deals  with  building  linkages  from  the  other  perspectives  into  the  objective 
perspective.  The  objective  perspective  contains  measures  that  directly  reflect  the  accomplishment  of 
objectives  identified  in  Section  4.2.2.  With  the  introduction  of  measures  into  each  perspective  of  the  BSC, 
the  following  analysis  should  take  place.  First,  the  questions  listed  in  Section  4.3.}  should  be  asked  about 
each  measure  included  in  the  perspective.  Subsequently,  relationships  between  success  in  this  measure 
and  success  in  other  measures  already  in  the  BSC  should  be  sought.  This  relationship  can  be  between  the 
new  measure  and  a  measure  in  the  objective  perspective,  or  it  can  be  between  the  new  measure  and  a 
measure  elsewhere  in  the  BSC  that  has  already  been  linked  to  the  objective  perspective.  If  no  linkages  can 
be  identified,  the  measure  is  either  a  diagnostic  measure  or  has  no  reason  to  be  on  the  scorecard.  For  more 
information  on  distinguishing  between  diagnostic  and  strategic  measures,  see  Section  4.3.3. 

After  determining  how  the  strategy  is  reflected  in  the  measures,  a  hypothesis  should  be  made  about  their 
correlation.  The  hypothesis  is  a  prediction  of  how  improvements  in  each  perspective  will  lead  to  an 
improved  bottom  line  in  the  objective  perspective.  As  a  hypothetical  example,  an  increase  in  “departure 
reliability”  in  the  “Management  Perspective”  of  ten  percent  may  have  a  positive  impact  on  “sorties 
flown”  in  the  objective  perspective  of  three  sorties  more  per  month.  Each  hypothesis  should  be  tested  and 
revised  to  give  a  more  accurate  correlation  as  needed.  During  the  infancy  of  the  scorecard,  the  testing  and 
revision  should  occur  frequently,  possibly  every  quarter.  As  the  BSC  matures,  testing  and  revision  may 
occur  once  a  year  or  less.  Occasionally,  the  hypothesis  turns  out  to  be  false  and  the  hypothesized 
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correlation  does  not  exist.  Upon  determining  that  a  hypothesis  is  false,  it  should  be  eliminated  and 
replaced  with  a  new  strategically  aligned  hypothesis. 


As  a  hypothetical  example,  a  strategic  correlation  related  to  the  goal  of  increasing  air  superiority  is 
discussed  next  and  shown  in  Figure  4.3. 


Figure  4.3:  Hypothetical  Strategic  Correlation 


Suppose  the  strategy  for  air  superiority  is  an  increase  in  sortie  generation  often  percent  over  the  next  five 
years.  “Sorties  flown”  is  chosen  as  the  measure  to  describe  this  goal.  “Sorties  flown”  is  located  in  the 
objective  perspective  and  is  directly  affected  by  “departure  reliability”  in  the  “Management  Perspective.” 

It  is  hypothesized  that  an  increase  in  “departure  reliability”  will  directly  affect  sortie  generation.  In  order 
to  increase  “departure  reliability,”  “repeat  recurrences”  must  be  reduced.  As  the  cannibalization  of  parts 
reduces  mission  capable  planes  and  increases  wear  and  tear  on  parts,  it  could  have  an  adverse  effect  on 
“repeat  recurrences”;  therefore,  a  reduction  of  the  “CANN  rate”  will  help  achieve  fewer  “repeat 
recurrences.”  It  must  be  recognized  that  this  is  a  completely  hypothetical  example,  created  to  illustrate  the 
process  of  forming  strategic  correlations.  It  is  important  to  note  that  each  of  the  amounts  hypothesized 
should  be  based  upon  the  principles  of  effective  goal  forming  given  in  Section  4.2.1. 

4.3.3  Measure  Finalization 

The  pared-down  measures  can  now  be  assigned  to  the  actual  scorecard.  These  measures  were  selected 
based  upon  their  strategic  linkage  in  the  scorecard  and  accurately  depict  the  strategy  of  the  organization. 
At  the  conclusion  of  an  initial  BSC  development,  there  may  be  gaps  in  the  correlation  where  the 
correlation  does  not  continue  all  the  way  through  the  scorecard.  The  gaps  in  the  correlation  indicate  where 
strategic  planning  needs  to  include  other  aspects,  such  as  internal  enhancement  or  influencing  factors.  The 
BSC  bridges  the  strategic  gap  between  organizational  success  and  the  factors  that  influence  it.  After  any 
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gaps  in  the  correlation  are  filled,  there  may  be  other  measures  that  still  need  to  be  added.  These  are  called 
diagnostic  measures,  which  are  not  linked  to  strategy  but  are  still  important.  These  measures  describe  key 
operating  statuses.  The  scorecard  can  be  analogically  compared  to  a  car  dashboard.  The  speedometer, 
odometer,  and  tachometer  are  strategic  measures.  The  diagnostic  measures  are  the  low-gas  light,  engine- 
maintenance  light,  and  the  low-oil  pressure  light.  Diagnostic  measures  help  to  identify  problems  before 
they  become  serious. 

4.3.4  Ownership  Assignment 

This  is  an  essential  step  in  BSC  development.  Each  measure  must  have  an  owner  who  is  responsible  for 
tracking  the  measure.  The  owner  has  the  responsibility  to  provide  thorough  documentation  describing  the 
measure,  provide  reasoning  for  past  performance,  and  supply  other  information  that  will  help  others  to 
interpret  and  assess  the  measure.  Although  a  measure  may  appear  on  multiple  scorecards,  there  is  never 
more  than  one  owner.  The  use  of  multiple  scorecards  is  discussed  in  the  following  section. 

4.3.5  Scorecard  Cascade 

In  general,  the  first  BSC  developed  in  any  organization  is  a  high-level  BSC.  After  a  high-level  BSC  has 
been  created,  it  should  be  cascaded.  Cascading  is  a  form  of  subdivision  of  measures.  The  measures  on  a 
high-level  scorecard  are  often  comprised  of  other  lower-level  measures  or  aggregates.  Cascading  is  the 
structuring  of  these  lower-level  measures  into  lower-level  scorecards.  As  previously  noted,  each  person 
should  only  have  twenty  or  fewer  measures  to  monitor.  The  common  practice  in  cascading  a  scorecard  is 
to  have  each  manager  create  a  scorecard  tailored  to  his  or  her  responsibilities.  Their  scorecards  contain 
measures  that  aggregate  into  the  measures  on  the  higher-level  scorecard.  An  example  of  aggregate 
measures  would  be  wing  “sorties  flown,”  which  is  comprised  of  the  “sorties  flown”  in  each  squadron.  A 
hypothetical  example  of  a  cascaded  scorecard  would  be  each  squadron’s  measure  of  “sorties  flown”  feeds 
into  the  aggregate  measure  of  “sorties  flown”  on  the  wing  scorecard.  The  lower-level  scorecards  will 
focus  on  unit-specific  responsibilities.  The  measures  on  a  high-level  scorecard  are  generally  very  abstract 
while  the  measures  on  the  lower-level  scorecards  become  increasingly  more  concrete.  When  cascading 
scorecards,  it  is  important  to  put  only  the  measures  necessary  to  the  person  or  unit  who  uses  the  scorecard. 
A  scorecard  for  the  flightline  will  not  contain  measures  that  pertain  to  munitions;  however,  the  higher- 
level  scorecard  over  both  these  units  will  contain  measures  from  both  flightline  and  munitions. 

4.3.6  Review  and  Revise 

Periodically,  a  BSC  must  be  revised  and  updated.  This  review  and  revision  should  take  place  frequently 
during  the  infancy  of  the  BSC.  In  the  beginning,  reviews  should  take  place  every  quarter  and  continue 
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until  the  hypotheses  made  in  Section  4.3.2  have  been  validated.  As  the  BSC  matures,  reviews  can  be 
conducted  annually  or  as  strategic  planning  requires.  Basic  revisions  can  take  place  at  any  time.  These  can 
include  cascading  down  additional  scorecards  when  new  units  are  added  or  reorganized.  Strategic  reviews 
should  occur  on  the  completion  dates  of  milestones  or  after  any  change  in  organizational  strategy.  At  this 
time,  major  changes  can  be  made  to  the  scorecard,  such  as  strategy  changes,  using  different  measures,  or 
changing  targets  for  measures. 


28 


5  BSC  Field  Study 

Section  4:  BSC  Development  Guide  provides  comprehensive  guidelines  for  developing  a  BSC.  In  this 
section,  a  case  study  is  presented  that  describes  the  process  undertaken  by  our  team  to  validate  and 
exemplify  the  BSC  Development  Guide.  The  team  developed  a  preliminary  BSC  for  flightline  MX 
activities  within  an  AMU.  For  each  step  of  the  BSC  Development  Guide,  this  section  provides  an 
explanation  of  the  work  performed  by  the  team  and  description  of  how  the  USAF  contributed. 

5. 1  Groundwork  Stage 

5.1.1  Team  Selection 

As  was  noted  in  Section  4.1.1,  a  team  must  be  assembled  to  develop  a  BSC.  The  BSC  team  is  typically 
made  up  of  multiple  members  of  a  unit  who  are  intricately  involved  in  the  development  process.  Since 
this  is  an  external  project,  the  team  was  created  in  a  unique  manner.  The  team  was  organized  according  to 
the  expertise  of  University  of  Arkansas  faculty  and  student  research  assistants.  Because  of  the  external 
nature  of  the  project,  USAF  personnel  from  the  AFRL,  Hill  Air  Force  Base  (AFB),  and  Barksdale  AFB 
collaborated  with  the  team  throughout  the  project. 

5.1.2  Strategic  Framework 

Formulating  a  strategy  is  essential  to  providing  the  overall  goals  and  objectives  that  are  linked  to  the 
BSC.  The  team  investigated  the  important  goals  of  the  AMU  through  site  visits,  personal  interviews,  and 
review  of  USAF  documentation.  The  collected  information  shaped  a  strategy  for  the  BSC.  The  developed 
strategy  is  only  intended  to  validate  the  correlation  process  for  the  preliminary  scorecard.  Using  the 
knowledge  gained  throughout  the  data  collection  process,  the  team  formed  and  utilized  the  following 
statement  as  the  overall  strategy  for  our  preliminary  BSC:  “The  strategy  of  a  typical  AMU  is  to  work 
toward  improvement  of  combat  capability  by  meeting  key  maintenance  performance  indicators.” 

5.1.3  Data  Collection 

Data  collection  is  at  the  heart  of  a  successful  BSC.  Without  pertinent  and  accurate  data  about  the  process 
being  studied,  the  exhaustive  list  of  possible  measures  cannot  be  correctly  formulated.  In  producing  the 
preliminary  BSC,  the  team  gathered  multiple  sources  of  information.  The  team  started  by  analyzing  the 
initial  problem  statement  and  developing  questions  about  the  scope  of  the  project.  The  team  needed 
complete  knowledge  of  how  a  BSC  was  formulated  and  implemented.  An  extensive  literature  search  was 
conducted  on  documentation  about  the  BSC  approach  and  applications  of  this  approach  within  industry. 
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This  literature  was  important  because  it  helped  further  explain  the  necessary  methodology  for  building  a 
BSC  and  organize  the  team’s  thoughts  on  how  to  apply  the  BSC  specifically  to  the  USAF.  Also,  in  order 
to  best  develop  the  BSC  Development  Guide,  the  team  needed  a  vast  amount  of  information  about  the 
AMU.  The  information  provided  by  the  USAF  included  specific  sources  for  measures  and  their  targets 
and  a  general  overview  of  the  command  hierarchy  within  the  USAF  MX  organization.  This  information 
was  provided  through  multiple  sources  discussed  in  the  following  sections. 

As  discussed  in  Section  4.1.3,  organizing  collected  data  is  vital  to  the  development  process.  To  maintain 
the  integrity  and  security  of  the  collected  information,  all  documents  were  kept  in  a  secure  location  with 
immediate  access  by  team  members. 

5.1. 3.1  Process  Data 

As  stated  in  Section  4. 1.3.1,  process  data  is  used  to  describe  what  actually  occurs  during  the  process  for 
which  the  BSC  is  being  developed.  Gathering  process  data  includes: 

1 .  Identifying  the  processes  within  the  system 

2.  Collecting  any  documentation  about  the  current  processes 

3.  Determining  the  chain  of  command  for  each  process 

4.  Documenting  the  processes 

Members  of  the  AFRL  team  visited  the  University  of  Arkansas  campus  to  conduct  a  brief  tutorial  of  high- 
level  Air  Force  operations.  From  this  tutorial,  the  need  for  the  team  to  see  and  collect  information 
concerning  the  flightline  process  was  confirmed.  The  team  visited  Hill  AFB  in  Ogden,  Utah  to  collect 
information  concerning  the  USAF  general  flightline  operations.  The  team  was  able  to  observe  the  entire 
flightline  process  from  the  aircraft  launch  through  all  the  processes  undertaken  after  its  landing  to  prepare 
it  for  another  launch.  The  team  conducted  briefings  with  individuals  concerning  supply,  scheduling,  and 
maintenance.  From  this  visit,  the  team  took  away  a  better  understanding  of  the  overall  process' and  began 
identifying  key  measures  used  by  the  USAF  for  measuring  performance  of  the  flightline  MX  activities. 

During  the  visit,  the  team  identified  documents  that  could  be  helpful  in  identifying  performance  measures. 
The  combination  of  site  visit  notes  and  USAF  documents  helped  the  team  begin  building  an  exhaustive 
list  of  performance  measures.  At  this  point,  the  majority  of  the  process  data  had  been  collected  and  the 
team  was  ready  to  collect  the  strategic  data. 
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5.1. 3.2  Strategic  Data 

Strategic  data  is  information  that  can  be  used  to  explain  the  mission  statement,  vision  statement,  and  core 
values  of  an  AMU  and  their  daily  implementation.  This  strategic  information  is  imperative  to  the 
formulation  of  the  BSC.  The  analysis  of  the  process  data  collected  from  the  first  site  visit  allowed  the 
team  to  focus  the  project’s  scope  and  pointed  out  the  need  for  strategic  data.  Process  data  provided 
information  about  the  current  performance  measures  in  use  but  not  the  strategy  that  the  USAF  uses  to 
govern  these  measures.  The  team  understood  the  basics  of  the  flightline  but  needed  to  understand  more 
about  the  strategic  focus  of  an  AMU.  A  second  site  visit  was  scheduled  to  Barksdale  AFB  in  Bossier  City, 
Louisiana.  The  main  focus  of  the  Barksdale  visit  was  to  answer  questions  concerning  the  strategic 
approach  of  the  AMU.  Such  questions  included: 

♦  Who  the  customers  for  the  preliminary  BSC  are 

♦  What  reports  are  being  used  to  measure  performance  that  the  team  might  not  already  have 

♦  What  are  the  most  important  measures  used  to  assess  performance 

The  team  conducted  interviews  with  USAF  personnel  including  the  MX  Group  Commander,  the  MX 
Squadron  Commander,  the  MX  Squadron  Chief  Enlisted  Manager,  and  the  AMU  Production  Supervisor 
(Pro  Super).  The  team’s  investigation  of  the  strategic  approach  sought  the  process  improvements  that 
would  lead  to  long-term  improvements  in  specific  areas  such  as  the  “mission-capable  rate”  and  the  total 
number  of  “sorties  flown.”  An  example  of  improvements  noticed  by  the  team  included  how  an  increase 
in  supply  chain  reliability  would  lead  to  a  decrease  in  the  frequency  of  cannibalization  actions  and  a 
decrease  in  the  MICAP  start-to-stop  time  durations.  The  team  noted  that  the  key  customers  for  the  team’s 
preliminary  BSC  are  the  operations  group,  individual  aircrews,  and  USAF  higher  command.  The  team 
was  introduced  to  the  9302  report  and  presentations  during  the  weekly  planning  meetings.  From  this 
information,  the  team  discovered  new  possible  measures  and  solidified  the  importance  of  others.  Some  of 
the  measures  that  stood  out  from  the  9302  forms  were  the  “CANN  rate,”  “maintenance  hours  per  flying 
hour,”  “PMCM”  and  “TNMCM.”  AFRL  personnel  provided  additional  assistance  by  arranging  the  site 
visit,  serving  as  base  guides  during  the  visit,  and  answering  post-visit  questions. 

5.1 .3.3  Reference  Materials 

As  stated  in  Section  4. 1.3. 3,  reference  materials  are  published  documents  that  contain  information  on 
relevant  processes  and  their  performance  measurement.  The  team  used  documentation  provided  by  the 
Air  Force  to  enhance  the  understanding  of  the  operations  under  observation.  The  reference  materials 
provided  by  the  Air  Force  included  training  and  instruction  guides  (ACCI  21-165,  2000;  ACCI  21-118, 
2002;  AFI  21-101  ACCSUP1-INT,  2003;  AFI  36-2201v3,  2002;  AFI  36-2232,  1999;  HOF  reports,  2003; 
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9302  forms,  2002),  and  the  Metrics  Handbook  for  Maintenance  Leaders  (AFLMA,  2001 )  to  collect 
process  and  strategic  data.  Because  the  team  relied  upon  the  USAF  for  the  majority  of  the  information 
about  its  operations,  reference  materials  were  collected  along  with  other  data  throughout  the  duration  of 
the  project. 

5.2  Design  Stage 

5.2.1  Goal  Development 

In  Section  4.2.1,  goal  development  is  described  as  the  process  of  turning  a  developed  strategy  into 
attainable  long-term  goals.  This  stage  is  invaluable  during  the  process  of  developing  a  working  BSC. 
Because  of  the  external  nature  of  this  project,  the  in-depth  knowledge  required  to  formulate  long-term 
goals  for  the  AMU  was  not  available;  thus,  the  team  developed  these  goals  based  upon  assumptions  made 
about  the  strategy  of  an  AMU.  The  main  goals  identified  by  the  team  were  “to  increase  the  total  number 
of  sorties  flown  in  a  given  period  of  time”  and  “to  increase  the  mission-capable  rate.” 

5.2.2  Objective  Identification 

Objective  identification  is  described  in  Section  4.2.2  as  the  conversion  of  strategies  and  long-term  goals 
into  specific  timelines  and  events.  For  the  team,  this  process  consisted  of  recognizing  some  of  the 
measures  that  should  be  included  in  the  objective  perspective,  because  this  is  the  perspective  that  drives 
the  BSC.  From  the  strategy  developed  in  Section  5.1.2  above,  the  team  was  able  to  identify  a  few  key 
measures  to  be  placed  in  the  objective  perspective  such  as  “sorties  flown,”  “mission-capable  rate,”  and 
“maintenance  hours  per  flying  hour.” 

5.2.3  Perspectives  Determination 

The  perspectives  are  critical  components  of  a  functional  BSC.  Perspectives  provide  a  central  theme  for 
each  group  of  key  measures  incorporated  into  the  BSC.  The  underlying  strategy  should  be  evident 
throughout  the  selection  of  perspectives.  The  research  team  used  the  information  gathered  through  site 
visits  and  USAF  documentation  to  determine  the  core  competencies  that  drive  success  for  an  AMU. 
These  core  competencies  led  the  team  to  inspect  how  the  original  business  perspectives  developed  by 
Kaplan  and  Norton  (1996)  could  be  adapted  to  the  AMU’s  structure.  The  team  identified  the  following 
four  perspectives  that  were  inline  with  the  identified  strategy  and  goals  of  an  AMU  and  stayed  true  to  the 
BSC  development  process: 
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♦  Mission  Perspective 

♦  Management  Perspective 

♦  Internal  Enhancement  Perspective 

♦  Influencing  Factors  Perspective 

The  original  BSC  (Kaplan  and  Norton,  1996)  was  built  around  the  objective  perspective.  All 
improvements  should  have  a  positive  impact  on  some  aspect  of  this  perspective.  Taking  this  into  account, 
the  research  team  determined  that  this  perspective  should  reflect  the  completion  of  the  prime  objectives  of 
the  AMU:  increasing  the  ability  to  successfully  accomplish  missions  and  the  ability  to  deploy  as  rapidly 
as  possible.  This  resulted  in  the  development  of  the  Mission  Perspective.  The  Mission  Perspective  reflects 
the  productivity  of  a  unit  by  measuring  factors  such  as: 

♦  How  many  units  are  flown  in  a  given  period 
How  many  planes  are  ready  to  fly  at  a  given  time 

♦  How  efficiently  a  unit  uses  its  time  when  compared  to  its  productivity 

All  other  perspectives  should  in  some  way  improve  this  perspective.  If  they  fail  to  do  this,  they  should 
either  be  represented  in  the  Mission  Perspective,  or  they  are  not  worthy  enough  to  include  in  the  BSC. 

The  BSC  perspectives  can  and  should  be  used  to  represent  major  stakeholders.  One  of  the  major 
stakeholders  of  the  original  BSC  is  the  customer.  The  customer-related  measures  are  used  to  measure 
market  share,  customer  retention,  customer  acquisition,  customer  satisfaction,  and  customer  profitability. 
The  “customer”  is  not  necessarily  the  best  indicator  for  the  USAF  to  use.  The  major  stakeholders  in  the 
place  of  outside  customers  are  the  management  command.  It  is  the  perception  of  the  higher  command  that 
is  important.  Many  of  the  measures  they  observe  deal  with  efficiency  in  operating  within  pre-set  standards 
or  schedules.  This  perspective  also  reflects  the  quality  of  work  done.  As  stated  above,  each  perspective 
must  improve  aspects  of  the  Mission  Perspective.  The  Management  Perspective  indicates  the  timeliness 
of  work  in  the  Mission  Perspective. 

Other  major  stakeholders  in  the  USAF  are  supply  and  scheduling  personnel.  Effective  supply  and 
scheduling  encompass  a  vital  part  of  increasing  the  USAF’s  ability  to  accomplish  its  mission  successfully. 
Supply  is  an  aspect  of  operations  that  has  a  direct  impact  on  success,  but  is,  however,  typically  out  of  the 
unit’s  direct  control.  Scheduling  is  similar  to  supply  in  that  it  is  not  under  the  control  of  the  AMU.  It  was 
observed  that  while  measures  in  this  perspective  can  be  very  informative,  they  should  be  considered  witli 
care.  They  must  not  be  given  undue  weight.  For  this  reason,  the  team  developed  the  Influencing  Factors 
Perspective  to  include  these  stakeholders. 
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Learning  and  growth  are  two  of  the  greatest  assets  of  the  BSC.  Through  them,  internal  investments  in 
training,  capital  investments  in  equipment,  and  building  of  infrastructure  can  be  linked  to  financial  gains. 
The  Internal  Enhancement  Perspective  is  similar  to  this  in  that  it  reflects  investments  into  training  and 
internal  quality.  Many  times  it  can  link  indirectly  into  Mission  Perspective  through  the  Management 
Perspective  by  increased  adherence  to  or  exceeding  of  pre-set  standards.  It  can  also  link  directly  into  the 
Mission  Perspective  through  increased  capability  and  capacity. 

The  four  perspectives  developed  by  the  research  team — Mission  Perspective,  Management  Perspective, 
Internal  Enhancement  Perspective,  and  Influencing  Factors  Perspective — should  be  considered  a  starting 
point  to  developing  a  BSC  for  an  AMU.  These  perspectives  should  be  carefully  reviewed  to  assist  in 
developing  perspectives  for  BSCs  for  use  in  an  actual  AMU.  The  most  important  aspect  of  a  BSC  is  the 
actual  development  process.  Success  comes  through  identifying  the  key  stakeholders  and  competencies 
and  then  exploiting  them. 

5.2.4  Performance  Measure  Identification 

The  final  measures  included  in  a  BSC  are  a  collection  of  a  specific  entity’s  most  important  and  relevant 
performance  measures.  The  measures  included  in  the  final  BSC  are  chosen  through  an  extensive  process 
The  first  step  in  identifying  the  measures  is  to  list  all  possible  measures  available  for  examination,  and 
also  identify  new  potential  measures.  From  this  now-exhaustive  list,  the  selection  process  can  begin. 

The  team  used  the  collected  documentation  and  external  interviews  to  identify  a  comprehensive  list  of 
performance  measures.  The  team  primarily  used  three  resources  to  identify  these  measures:  the  9302 
forms  (2002),  the  HOF  reports  (2003),  and  the  Metrics  Handbook  for  Maintenance  Leaders  (AFLMA, 
2001).  Additional  measures  were  identified  by  a  team  member  in  an  interview  with  a  retired  USAF 
General.  The  team  sought  additional  measures  to  contribute  to  the  Internal  Enhancement  Perspective. 
Most  of  the  measures  collected  from  documentation  appeared  to  fit  well  within  the  other  three 
perspectives,  but  measures  for  the  Internal  Enhancement  Perspective  were  more  difficult  to  identify. 
Some  of  the  measures  identified  for  this  perspective  include  Equal  Employment  Opportunity  (EEO) 
complaints,  Inspector  General  (IG)  complaints,  and  Special  Experience  Identifiers. 


Each  listed  performance  measure  was  identified  by  the  team  as  a  measure  used  by  the  I ISAF  to  gauge  an 
AMU’s  execution  of  flightline  activities.  The  comprehensive  list  created  by  the  team  includes  the  name  of 
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the  measure  and  which,  if  any,  of  the  three  predominately  used  resources  the  measure  came  from.  The 
exhaustive  list  of  performance  measures  is  provided  in  Appendix  2. 

5.2.5  Measure  to  Family  Assignment 

Once  all  measures  are  compiled,  families  are  established  and  measures  are  assigned  to  a  specific  family. 
Families  are  used  to  organize  the  list  of  identified  measures  into  groups  based  upon  the  characteristics  or 
objectives  of  each  measure.  The  team  reviewed  the  exhaustive  list  of  performance  measures  and  assessed 
the  similarities  among  the  measures.  The  similarities  that  surfaced  became  the  basis  of  the  created 
families.  The  family  names  and  a  description  of  each  family  are  discussed  in  Table  5.1. 


Table  5.1:  BSC  Family  Descriptions 


Productivity 

These  are  all  bottom-line  measures  that  show  how  well  a  unit  is  accomplishing  its 
mission  by  quantity  and  quality  of  its  deliverables. 

Supply 

These  are  all  outside  supply  measures.  These  are  indirect  factors  that  must  be 
controlled  in  order  to  accomplish  mission  objectives. 

Timeliness 

These  are  measures  that  reflect  how  well  a  unit  is  accomplishing  its  tasks  on  time.  Its 
measures  also  reflect  how  well  a  unit  completes  repairs  compared  to  the  average 
time  required  for  these  repairs  USAF-wide. 

Excellence 

These  measures  reflect  the  quality  of  work  accomplished  by  a  unit. 

Growth 

These  are  measures  that  reflect  the  morale  of  a  unit.  They  also  reflect  the  behavior  of 
its  members  and  show  their  efforts  toward  self-improvement. 

Scheduling 

These  are  additional  measures  that  show  outside  influences  on  an  AMU’s  work.  They 
show  how  well  an  AMU  complies  with  the  flight  schedules  and  what  might  be  outside 
their  control. 

Each  measure  was  then  assigned  to  a  family  based  upon  its  association  with  the  most  appropriate  family 
description.  The  assignment  of  each  measure  and  its  corresponding  family  is  provided  in  Appendix  3. 

Next  the  team  assessed  the  relationship  of  each  family  to  the  four  perspectives.  A  perspective  can  have 
more  than  one  family,  but  each  family  cannot  be  assigned  to  more  than  one  perspective.  Table  5.2 
provides  the  four  perspectives  created  and  each  perspective’s  corresponding  family  or  families.  The 
families  serve  as  attributes  for  each  of  the  perspectives. 
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Table  5.2:  Perspective  to  Family  Relationship 


Perspective 

Family 

Mission 

Productivity 

Influencing  Factors 

Scheduling 

Supply 

Management 

Timeliness 

Internal  Enhancement 

Excellence 

Growth 

5.3  Finalization  Stage 

5.3.1  Measure  Assignment  to  the  BSC 

Assigning  measures  to  the  BSC  is  the  culmination  of  all  the  previous  steps.  The  perspectives  and  families 
have  been  defined,  and  now  is  the  time  to  begin  selecting  the  measures  that  best  represent  the  overall 
strategy  of  the  AMU.  When  the  team  completed  the  exhaustive  list  of  measures,  the  number  of  measures 
accumulated  was  close  to  90.  A  typical  BSC  contains  between  20  -  25  measures.  The  candidate  measures 
were  evaluated  and  eliminated  according  to  their  perceived  value  toward  achieving  the  overall  strategy  of 
the  AMU  and  the  importance  of  each  measure  in  accordance  to  its  appearance  in  USAF  documentation. 
This  process  was  used  to  create  a  reduced  list  of  measures  that  were  likely  candidates  for  inclusion  in  the 
BSC. 

After  this  initial  screening,  the  remaining  measures  were  evaluated  based  upon  the  Caplice  and  Sheffi 
(1994)  criteria  found  in  Table  4.1.  By  evaluating  the  measures  based  upon  these  trade-offs,  the  team  was 
able  to  cultivate  a  smaller  pool  of  candidates  for  use  in  the  BSC.  Assigning  measures  to  the  BSC  is  not 
the  process  that  produces  the  final  set  of  measures.  The  intention  of  the  process  is  to  establish  a  set  of 
measures  that  could  be  used  in  the  BSC  if  they  succeed  through  the  final  stages  of  the  development 
process. 
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5.3.2  Correlation  Determination 

Forming  linkages  between  measures  in  the  BSC  creates  cohesiveness  between  the  perspectives  that 
allows  the  organization  to  maximize  the  use  of  the  BSC.  When  strategies  are  developed,  correlations  can 
be  produced  within  the  measures  of  the  BSC.  Given  the  external  nature  of  this  research  and  the  dynamic 
nature  of  the  USAF,  the  team  could  only  hypothesize  possible  strategies  for  Air  Force  success  because  of 
the  lack  of  internal  process  knowledge.  The  hypothesized  strategies  were  used  to  develop  an  example 
correlation  to  test  the  team’s  example  scorecard.  The  example  correlation  was  based  upon  the  team’s 
experience  throughout  the  project,  and  was  explained  in  Section  4.3.2.  This  correlation  allowed  the  team 
to  solidify  some  of  the  measures  to  be  put  into  the  final  BSC. 

Correlations  should  be  developed  using  the  firm  foundation  of  strategic  knowledge  from  within  an  AMU. 
From  an  internal  perspective,  correlations  should  be  extensively  evaluated  before  finalizing  the  measures 
in  the  BSC.  The  correlations  between  various  measures  could  greatly  affect  the  success  of  the  BSC.  The 
purpose  of  the  BSC  is  to  tie  multiple  facets  of  an  organization  together;  correlations  exhibit  those  ties. 

5.3.3  Measure  Finalization 

The  final  measures  to  be  included  in  the  BSC  will  represent  the  perspectives  comprehensively  to  ensure 
that  an  accurate  reading  of  the  organization  as  a  whole  can  be  taken.  Each  measure’s  definition  or 
function  is  very  important  when  settling  on  the  final  measures  (Table  5.3).  After  the  correlations  are 
produced,  there  may  be  perspectives  with  fewer  measures  than  anticipated.  As  was  discussed  in  Section 
4.3.3,  a  BSC  should  contain  two  different  types  of  measures:  strategic  and  diagnostic.  Strategic  measures 
define  an  organizational  strategy.  The  measures  remaining  to  be  assigned  after  the  determination  of 
correlation  are  likely  to  be  diagnostic.  Diagnostic  measures  monitor  whether  a  process  stays  in  control. 
Diagnostic  measures  fill  gaps  as  the  measures  of  the  BSC  are  finalized. 

Because  of  the  external  nature  of  the  project,  the  team  selected  some  of  the  final  measures  for  the 
preliminary  BSC  based  more  upon  the  collective  knowledge  of  the  team  than  on  the  correlations 
produced.  The  team  selected  a  small  portion  of  the  strategic  measures  from  the  correlations  produced  and 
the  other  portion  based  upon  the  assumptions  made  about  the  strategic  goals  of  an  AMU.  The  team  filled 
in  the  gaps  with  diagnostic  measures  such  as  “CANN  rate,”  “TNMCM,”  and  “total  abort  rate.”  For  a  final 
AMU  BSC,  more  measures  should  be  finalized  through  the  study  of  correlations.  This  would  solidify  the 
strategic  linkages  within  the  BSC  and  therefore  increase  the  resulting  benefits.  The  preliminary  BSC 
developed  by  the  team  is  shown  in  Figure  5.1  and  the  preliminary  definitions  are  shown  in  Table  5.3. 
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Mission  Perspective 

-  Long-term  ability  to  improve  combat  capability 


Influencing  Factors  Perspective 

-  Success  with  collaborators  and  functional 
processes 


♦  CANN  rate 

♦  MICAP  fill  rates 

♦  MSE 

♦  TNMCS 


Figure  5.1:  Preliminary  BSC  for  AMU  Fiightline  MX  Activities 


Internal  Enhancement  Perspective 

-  Necessity  for  innovation  and  growth 

♦  CND  rate 

♦  RR  rate 

♦  Special  Experience  Identifiers 

♦  Total  abort  rate 

♦  Training  schedule  adherence 

♦  Upgrade  Training 

♦  Unit  average  technical  skill  level 


Management  Perspective 

-  Achievement  of  customer  service 

♦  Adherence  to  OPS/MXS  schedule 

♦  DD  rate  AWM 

♦  Total  Maintenance  deviations 

♦  4-hour  fix  rate 

♦  8-hour  fix  rate 

♦  12-hour  fix  rate 

♦  Days  in  Phase/ISO 


♦  Maintenance  hours  per  flying  hour 

♦  MC  rate 

♦  PMCM 

♦  Sorties  flown 

♦  TNMCM 


Table  5.3:  Definitions  of  Preliminary  BSC  Measures 


Mission  Perspective 

Maintenance  hours  per  flying  hour 

The  total  number  of  maintenance  hours  performed 
versus  the  total  number  of  flying  hours 

MC  Rate 

The  total  number  of  mission-capable  hours  (fully  or 
partially)  versus  the  total  number  of  possessed  hours 

PMCM 

The  number  of  possessed  hours  an  aircraft  is  partially 

mission-capable  due  to  needed  maintenance 
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Mission  Perspective 


Sorties  flown 

The  number  of  sorties  flown  in  a  specified  period  of 
time  (week,  month,  year) 

TNMCM 

The  number  of  possessed  hours  an  aircraft  is  totally 
not  mission-capable  due  to  maintenance 

influencing  Factors  Perspective 

CANN  rate 

The  number  of  cannibalization  actions  versus  the 
number  of  sorties  flown  in  a  given  time  period  (week, 
month,  year) 

MICAP  fill  rate 

The  time  duration  or  how  long  the  fill  process  takes  for 
a  Mission-Impaired  Capability  Awaiting  Parts  aircraft  . 

MSE 

The  number  of  maintenance  actions  started  versus  the 
number  of  maintenance  actions  scheduled 

TNMCS 

The  number  of  possessed  hours  an  aircraft  is  totally 
not  mission-capable  due  to  supply 

Management  Perspective 

Adherence  to  OPS/MXS  schedule 

The  number  of  deviations  from  the  OPS  or  MXS 
schedule  committed 

DD  rate  AW\M 

The  average  number  of  deferred  discrepancies  due  to 
maintenance  in  a  given  time  period  (week,  month, 
year) 

Total  maintenance  deviation 

The  total  number  of  deviations  from  the  maintenance 
schedule 

4-hour  fix  rate 

The  percentage  of  aircraft  that  return  with  a  Code  3 
break  and  are  returned  to  MC  status  within  8  hours 

8-hour  fix  rate 

The  percentage  of  aircraft  that  return  with  a  Code  3 
break  and  are  returned  to  MC  status  within  4  hours 

12-hour  fix  rate 

The  percentage  of  aircraft  that  return  with  a  Code  3 
break  and  are  returned  to  MC  status  within  12  hours 

Internal  Enhancement  Perspective 

CND  rate 

The  total  number  of  in-flight  discrepancies  that  could 
not  be  duplicated  by  maintenance  personnel  versus 
the  total  number  of  in-flight  discrepancies 

RR  rate 

The  number  of  fixed  problems  that  reoccur  versus  the 
total  number  of  fixed  problems 
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Mission  Perspective 

Special  experience  identifier 

Special  hands-on  training  or  experience  an  individual 
receives  and  is  given  credit  for 

Total  abort  rate 

The  total  number  of  sorties  aborted  versus  the  total 
number  of  sorties  attempted 

Training  schedule  adherence 

The  number  of  deviations  from  the  training  schedule 
committed 

Upgrade  training 

The  status  of  an  individual  concerning  the  next 
upgrade  in  level  of  maintenance  training 

Unit  average  technical  skill  level 

The  cumulative  skill  level  of  personnel  in  a  unit  per  the 
number  of  personnel  in  a  unit 

Because  the  BSC  was  externally  produced,  validation  was  an  important  step  in  the  case  study.  The  team 
created  a  questionnaire  (presented  in  Appendix  4)  that  was  administered  by  AFRL  personnel  to  attendees 
at  the  2003  LOA  National  Conference.  This  conference  was  held  13-16  October  03  in  Oklahoma  City, 
Oklahoma.  The  questionnaire  sought  the  expertise  of  logistics  personnel  in  ranking  the  criticality  of  the 
measures  on  the  preliminary  BSC.  Additional  information  about  the  analysis  and  results  of  the 
questionnaire  is  provided  in  Section  6. 

5.3.4  Ownership  Assignment 

Assigning  ownership  of  measures  to  specific  individuals  is  important  to  the  success  of  a  BSC  because  a 
single  person  cannot  constantly  monitor  or  be  held  accountable  for  all  of  the  BSC  measures.  Because  the 
team  is  external  and  the  BSC  is  not  being  implemented  at  this  time,  ownership  assignment  was  not 
undertaken.  Ownership  to  particular  measures  would  be  assigned  by  the  individual  in  charge  of  the  entire 
BSC.  Since  multiple  BSCs  are  possible  within  one  wing  or  even  one  squadron,  one  individual  may  own 
an  entire  BSC  and  also  use  their  BSC  as  a  single  measure  for  a  higher-level  BSC. 

5.3.5  Scorecard  Cascade 

Scorecard  cascading  is  outside  the  scope  of  this  project.  The  team  did  develop  one  suggestion  of  a 
possible  opportunity  for  cascading.  The  daily  and  weekly  meeting  hierarchy  would  be  an  excellent  way  to 
cascade  the  BSC.  The  commander  would  have  the  high-level  BSC  developed  by  the  project  team.  Each 
sub-commander  would  have  a  scorecard  developed  with  measures  cascaded  down  from  measures  in  the 
commander’s  scorecard.  The  aggregation  of  each  of  the  sub-commanders’  BSCs  would  form  the 
commander’s  BSC. 
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5.3.6  Review  and  Revision 

Review  and  revision  is  key  to  the  continued  success  of  a  BSC.  By  reviewing  the  BSC  at  scheduled 
intervals,  the  opportunity  for  continuous  improvement  exists.  Within  the  context  of  a  functioning  BSC, 
the  internal  team  would  use  the  review  time  as  constructive  criticism  of  the  current  BSC.  This  is  both 
productive  and  healthy  for  an  organization.  Continuous  improvement  is  imperative  to  the  survival  of 
efficient  practices  and  success  as  an  organization. 
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6  BSC  Validation 

The  questionnaire  discussed  in  Section  5.3.3  and  presented  in  Appendix  4  was  used  to  obtain  expert 
opinions  about  the  hierarchy  of  the  perspectives  and  measures  included  in  the  preliminary  BSC.  The 
purpose  of  this  questionnaire  was  to  see  if  logistics  personnel  believed  that  there  were  more  important 
measures  that  should  be  added  or  current  measures  that  should  be  omitted.  The  questionnaire  requested 
each  respondent  to  rank  each  of  the  four  perspectives  in  decreasing  criticality  and  rank  each  measure 
within  the  four  perspectives  by  decreasing  criticality  within  their  perspective.  The  rankings  used  an 
increasing  numeric  scale  to  portray  relative  criticality:  the  lower  the  number,  the  higher  the  criticality, 
with  1  being  the  most  critical.  The  questionnaire  also  requested  that  the  respondents  suggest  any  omitted 
measures  that  they  deemed  pertinent. 

Thirty-six  respondents  completed  the  anonymous  questionnaire.  Responses  from  each  questionnaire  were 
entered  into  a  Microsoft®  Excel  spreadsheet.  This  Respondent  Raw  Data  Spreadsheet  is  presented  in 
Appendix  5.  The  spreadsheet  is  divided  into  three  sections  corresponding  to  the  information  requested  on 
the  questionnaire.  The  first  section  displays  how  each  individual  ranked  the  four  perspectives  on  a  scale 
of  one  to  four,  with  one  being  the  most  critical  and  four  being  the  least.  The  second  section  contained  four 
parts,  displaying  how  each  individual  ranked  the  measures  corresponding  to  each  of  the  four  perspectives. 
The  third  section  displayed  the  job  classification  of  the  respondent. 

The  questionnaires  were  examined  for  completeness  and  usefulness  and  a  set  of  criteria  was  developed  to 
determine  possible  outliers.  Outliers  were  questionnaires  that  did  not  fully  conform  to  the  guidelines  of 
the  questionnaire.  These  outlier  questionnaires  were  removed  from  the  study,  resulting  in  26  viable 
respondents.  The  discarded  questionnaires  were  2,  3,  4,  9,  14,  16,25,  27,  34,  and  35  as  shown  by  the 
shaded  rows  of  the  Respondent  Raw  Data  Spreadsheet.  An  example  of  an  outlier  questionnaire  is  one 
where  the  respondent  combined  two  of  the  perspectives  into  one  and  ranked  them  using  their  own 
method.  There  were  also  questionnaires  with  minor  problems  such  as  only  three  of  the  four  perspectives 
being  ranked  or  the  measures  being  ranked  but  not  the  perspectives  that  were  also  discarded. 

Next,  a  count  of  the  times  each  perspective  and  each  measure  within  each  perspective  was  assigned  a 
particular  ranking  of  1, 2,  3,  etc.  was  computed.  The  corresponding  percentage  represented  the  number  of 

times  each  perspective  or  measure  was  given  that  ranking  out  of  the  total  number  of  questionnaires.  The 

percentages  allowed  the  data  to  be  conceptualized  in  a  more  meaningful  manner. 
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In  addition  to  the  comprehensive  results  of  the  respondents  as  a  whole,  the  results  were  categorized  by 
respondent  job  function  level  where  adequate  sample  size  within  the  category  permitted,  specifically 
group-level  maintenance  supervision/staff  and  squadron-level  maintenance  supervision/staff.  Of  the  26 
questionnaires  used  for  analysis,  23  of  them  fell  into  one  of  these  two  job  function  levels.  The  analysis 
was  repeated  for  each  group  individually,  allowing  for  observation  of  how  the  two  levels  of  job  function 
may  differ  in  their  view  of  the  criticality  of  the  perspectives  or  measures. 

The  analyzed  data  from  the  various  groupings  is  presented  in  a  graphical  format  in  the  following  sections. 
The  graphs  are  stacked  bar  graphs  w'ith  each  column  representing  a  specific  perspective  or  measure.  The 
bars  contain  the  percentages  of  each  ranking  (Is,  2s,  3s,  etc.)  that  were  obtained  by  each  perspective  or 
measure  within  its  perspective.  Separate  graphs  represent  the  criticality  of  the  four  perspectives  and  the 
criticality  of  each  measure  within  its  perspective.  Sections  6.1  and  6.2  respectively  present  the  analyzed 
results  from  all  questionnaires  (26)  and  the  questionnaires  categorized  by  job  function  level  including 
group-level  (10)  and  squadron-level  (13). 
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6.1  Comprehensive  Results 

Figure  6.1  presents  the  summary  of  the  criticality'  of  the  perspectives  from  all  26  viable  respondents.  The 
results  indicate  that  the  Mission  Perspective  is  the  most  critical  perspective  with  62%  of  the  respondents 
giving  it  the  top  ranking  (1).  From  the  results,  it  can  be  concluded  that  Management  Perspective  is  the 
next  most  critical  with  23%  of  the  respondents  ranking  it  as  the  most  critical  (1).  The  least  critical 
perspective  is  Internal  Enhancement  Perspective  with  62%  of  the  respondents  ranking  it  as  the  least 
critical  (4).  This  is  an  expected  finding,  as  this  perspective  contains  measures  that  are  least  tangible  and 
has  the  most  measures  that  are  not  currently  being  tracked  by  the  USAF. 


Perspectives 


Mission  Influencing  Factors  Management  Internal  Enhancement 


Figure  6.1:  Distribution  of  Responses  to  Perspectives 


44 


Figure  6.2  presents  the  criticality  of  the  Mission  Perspective  measures.  The  measures  were  ranked  on  a 
scale  from  1  to  5,  with  1  being  the  most  critical  measure.  The  results  clearly  indicate  the  “MC  rate”  is  the 
most  critical  measure  with  58%  of  the  individuals  ranking  it  as  the  most  critical  (1).  “Maintenance  hours 
per  flying  hour”  measure  is  the  second  most  critical  with  50%  of  the  respondents  ranking  it  as  first  or 
second  most  critical  (1  or  2).  It  is  obvious  by  the  graph  that  “PMCM”  is  the  least  critical  measure  with 
58%  of  the  respondents  ranking  as  the  least  critical  (5)  and  no  respondent  ranking  it  as  most  critical. 
Based  upon  these  findings,  “PMCM”  is  a  strong  candidate  for  removal  from  the  preliminary  BSC. 


Mission  Perspective 


n  no/. 


Maint.  Hours/  Fly  MC  Rate  PMCM  Sorties  Flown  TNMCM 

Hours 


Figure  6.2:  Distribution  of  Responses  to  the  Mission  Perspective  Measures 
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Figure  6.3  presents  the  criticality  of  the  Influencing  Factors  Perspective  measures.  The  measures  were 
ranked  on  a  scale  from  1  to  4,  with  1  being  the  most  critical  measure.  The  results  indicate  the  “MICAP  fill 
rate”  is  the  most  critical  measure  with  38%  of  the  individuals  ranking  it  as  the  most  critical  (1 ).  The 
ranking  of  the  other  measures  in  this  perspective  are  less  conclusive.  Interestingly,  “Maintenance 
Scheduling  Effectiveness”  has  the  next  highest  percentage  of  respondents  (31%)  ranking  it  as  most 
critical  (1)  and  the  highest  percentage  of  respondents  (50%)  ranking  it  as  least  critical  (4).  Respectively 
53%  and  42%  of  the  respondents  rank  “CANN  Rate”  and  “TNMCS”  as  most  (1)  or  second  most  (2) 
critical. 


Influencing  Factors  Perspective 


CANN  Rate  MICAP  Fill  Rates  Maint.  Scheduling  TNMCS 

Effectiveness 


Figure  6.3:  Distribution  of  Responses  to  the  Influencing  Factors  Perspective  Measures 


46 


Figure  6.4  presents  the  criticality  of  the  Management  Perspective  measures.  The  measures  were  ranked 
on  a  scale  from  1  to  7,  with  1  being  the  most  critical  measure.  The  results  strongly  indicate  the 
“Adherence  to  OPS/MXS  Schedule”  is  the  most  critical  measure  with  58%  of  the  respondents  ranking  it 
as  most  critical  (1).  The  next  most  critical  measure  is  “Total  Maintenance  Deviations”  with  39%  of  the 
respondents  ranking  it  as  most  (1)  or  second  most  (2)  critical.  The  “12-Hour  Fix  Rate”  was  deemed  more 
critical  than  the  “4-Hour  Fix  Rate”  and  “8-Hour  Fix  Rate.”  Potential  measures  for  removal  from  the  BSC 
are  “DD  Rate  AWM”  and  “4-Hour  Fix  Rate.” 


Management  Perspective 


Adherence  to  DD  Rate  Total  4-Hour  Fix  8-Hour  Fix  12-Hour  Fix  Days  in 
OPS/MXS  AWM  Maintenance  Rate  Rate  Rate  Phase/ISO 

Schedule  Deviations 


Figure  6.4:  Distribution  of  Responses  to  the  Management  Perspective  Measures 
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Figure  6.5  presents  the  criticality  of  the  Internal  Enhancement  Perspective  measures.  The  measures  were 
ranked  on  a  scale  from  1  to  7,  with  1  being  the  most  critical  measure.  The  measure  ranking  results  of  this 
perspective  is  less  conclusive.  The  results  indicate  that  “CND  rate,”  Repeat-Recur  (RR)  Rate,”  Upgrade 
Training,”  and  “Unit  Average  Technical  Skill  Level”  are  the  most  critical  measures  with  the  greatest 
number  of  respondents  ranking  these  measures  as  most  (1)  or  second  most  (2)  critical.  “Special 
Experience  Identifiers”  was  ranked  the  least  critical  (7)  with  50%. 


Internal  Enhancement  Perspective 


CND  Rate  Repeat-  Special  TotalAbort  Training  Upgrade  UnitAverage 
Recur  (RR)  Experience  Rate  Schedule  Training  Technical 
Rate  Identifiers  Adherence  Skill  Level 


Figure  6.5:  Distribution  of  Responses  to  the  Internal  Enhancement  Perspective  Measures 


6.2  Level  of  Job  Function  Results 

The  graphs  for  respondent  rankings  by  job  function  level  are  presented  in  this  section.  The  majority  of  the 
observations  are  similar  to  those  made  in  the  combined  results  presented  in  Section  6.1.  Important 
findings  that  differ  between  the  two  job  function  levels  are  noted. 
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6.2.1  Perspective  Rankings 

Figure  6.6  and  Figure  6. 7  present  the  criticality  of  the  perspectives  from  the  group-level  and  squadron- 
level  respondents  respectively.  Both  sets  of  ranking  results  are  similar  to  the  combined  ranking  shown  in 
Figure  6.1  except  for  the  least  critical  perspective.  Observation  of  Figure  6.6  shows  that  Influencing 
Factors  Perspective  is  the  least  important  perspective  for  the  group-level  respondents,  while  Figure  6. 7 
indicates  that  the  Internal  Enhancement  Perspective  is  least  critical  for  the  squadron-level  respondents. 
This  result  is  interesting  and  supports  the  cascading  of  different  BSCs  for  differing  levels  of  the  USAF. 


Perspective  (Group  Level) 


Mission  Influencing  Factors  Management  Internal 

Enhancement 


Figure  6.6:  Distribution  of  Group-Level  Responses  to  Perspectives 
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Perspectives  (Squadron  Level) 


Mission  Influencing  Factors  Management  Internal  Enhancement 


Figure  6.7:  Distribution  of  Squadron-Level  Responses  to  Perspectives 

6.2.2  Measure  Rankings  for  the  Mission  Perspective 

Figure  6.8  and  Figure  6.9  present  the  criticality  of  the  measures  within  the  Mission  Perspective  from  the 
group-level  and  squadron-level  respondents  respectively.  Both  sets  of  ranking  results  are  similar  to  the 
combined  ranking  shown  in  Figure  6.2. 
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Mission  Perspective  (Group  Level) 
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Figure  6.8:  Distribution  of  Group-Level  Responses  to  the  Mission  Perspective  Measures 


Mission  Perspective  (Squadron  Level) 


Figure  6.9:  Distribution  of  Squadron-Level  Responses  to  the  Mission  Perspective 

6.2.3  Measure  Rankings  for  the  Influencing  Factors  Perspective 

Figure  6.10  and  Figure  6.11  present  the  criticality  of  the  measures  within  the  Influencing  Factors 
Perspective  from  the  group-level  and  squadron-level  respondents  respectively.  Both  sets  of  ranking 
results  are  similar  to  the  combined  ranking  shown  in  Figure  6.3  except  for  the  ranking  of  the  MSE.  This 
measure  has  very  low  criticality  for  the  group-level  with  80%  of  the  group-level  respondents  ranking  it 
least  critical  (4),  but  has  the  second-highest  criticality  for  the  squadron-level  respondents. 
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Influencing  Factors  Perspective  (Group  Level) 


CANN  Rate  MICAP  Fill  Rate  Maint.  Scheduling  TNMCS 

Effectiveness 


Figure  6.10:  Distribution  of  Group-Level  Responses  to  the  Influencing  Factors  Perspective 

Measures 
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Influencing  Factors  (Squadron  Level) 
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Figure  6.11:  Distribution  of  Squadron-Level  Responses  to  the  Influencing  Factors  Perspective 

Measures 

6.2.4  Measure  Rankings  for  the  Management  Perspective 

Figure  6. 12  and  Figure  6. 13  present  the  criticality  of  the  measures  within  the  Management  Perspective 
from  the  group-level  and  squadron-level  respondents  respectively.  Both  sets  of  ranking  results  are  similar 
to  the  combined  ranking  shown  in  Figure  6.4. 


Adherence  to  DDRateAWM  Total  4-Hour  Fix  8-Hour  Fix  12-Hour  Fix  Days  in 
OPS/MXS  Maintenance  Rate  Rate  Rate  Ftiase/ISO 

Schedule  Deviations 


Figure  6.12:  Distribution  of  Group-Level  Responses  to  the  Management  Perspective  Measures 


Management  Perspective  (Squadron  Level) 


Adherence  to  DD  Rate  Total  4-Hour  Fix  8-Hour  Fix  12-Hour  Fix  Days  in 

OPS/MXS  AWM  Maintenance  Rate  Rate  Rate  Phase/ISO 

Schedule  Deviations 


Figure  6.13:  Distribution  of  Squadron-Level  Responses  to  the  Management  Perspective  Measures 

6.2.5  Measure  Rankings  for  the  Internal  Enhancement  Perspective 

Figure  6.14  and  Figure  6.15  present  the  criticality  of  the  measures  within  the  Internal  Enhancement 
Perspective  from  the  group-level  and  squadron-level  respondents  respectively.  The  sets  of  ranking  results 
differ  from  the  combined  ranking  shown  in  Figure  6.5  in  their  assignment  of  the  most  critical  measures. 
Figure  6.14  shows  that  the  most  critical  measures  for  the  group-level  respondents  are  “Repeat-Recur 
(RR)  Rate”  and  “Unit  Average  Technical  Skill  Level”  similar  to  the  combined  results  in  Figure  6.5.  The 
squadron-level  results  shown  in  Figure  6.15  indicate  the  most  critical  measures  as  “Total  Abort  Rate”  and 
“Upgrade  Training.”  These  differences  again  support  the  use  of  different  BSCs  for  differing  levels  of  job 
functions. 
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Management  Perspective  (Squadron  Level) 


Adherence  to  DD  Rate  Total  4-Hour  Fix  8-Hour  Fix  12-Hour  Fix  Days  in 

OPS/MXS  AWM  Maintenance  Rate  Rate  Rate  Phase/ISO 

Schedule  Deviations 


Figure  6.15:  Distribution  of  Squadron-Level  Responses  to  the  Internal  Enhancement  Perspective 

Measures 
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7  BSC  Software  Package  Review 


7.1  BSC  Software 

This  section  discusses  BSC  software,  including  standards  for  the  selection  and  a  detailed  review  of  three 
software  packages.  Through  the  BSC,  strategies  are  mapped  and  monitored.  The  BSC  allows  management 
to  keep  track  of  lagging  and  leading  indicators  to  indicate  the  success  and  failure  of  these  strategies.  The 
BSC  facilitates  cascading  of  these  strategies  throughout  an  entire  organization.  In  order  to  successfully 
accomplish  this,  many  different  variables  must  be  monitored  and  communicated  to  a  variety  of 
stakeholders.  It  is  useful  for  these  stakeholders  to  have  these  variables  visualized  in  the  format  of  a  BSC. 
The  management  and  presentation  of  this  extensive  information  is  the  primary  reason  BSC  software 
packages  have  been  developed. 

7.2  Standards  for  the  Software 

There  are  many  BSC  software  packages  available  on  the  market,  but  limited  standards  by  which  to 
compare  them.  The  Balanced  Scorecard  Collaborative,  Inc.  (BSCol)  is  a  firm  that  was  founded  by  Kaplan 
and  Norton,  creators  of  the  BSC,  to  increase  awareness  and  use  of  the  BSC.  The  BSCol  developed  a  set  of 
standards  to  assist  end-users  and  developers  of  BSC  software  packages  in  assuring  that  their  software 
packages  make  full  use  of  the  BSC.  The  complete  functional  standards  documentation  provided  by  the 
BSCol  is  provided  in  Appendix  6.  The  stated  purposes  of  these  standards  are: 

♦  To  provide  guidance  to  user  organizations  in  their  evaluation  and  selection  of  a  BSC  software 
package 

♦  To  help  define  development  guidelines  for  software  companies  seeking  to  support  the  Kaplan  and 
Norton  BSC  methodology 

7.3  BSCol  Functional  Standards 

The  BSCol  suggests  four  categories  of  features  that  should  be  included  in  the  BSC  software  packages: 

♦  Design 

♦  Strategic  Education  and  Communication 

♦  Business  Execution 

♦  Feedback  and  Learning 

The  DSCoI  oonaidoro  those  as  the  minimum  necessary  features,  and  many  software  packages  go  beyond 

them.  A  comprehensive  discussion  of  these  categories  is  located  in  Section  7.4. 
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7.3.1  Design 

The  BSCol  suggests  the  following  features  for  the  Design  category: 

♦  View  strategy  from  the  four  perspectives 

♦  Identify  strategic  objectives  for  each  perspective 

♦  Associate  measures  with  strategic  objectives 

♦  Link  measures  in  cause-and-effect  relationships 

♦  Assign  targets  to  measures 

♦  List  strategic  initiatives 

7.3.2  Strategic  Education  and  Communication 

The  BSCol  suggests  the  following  features  for  the  Strategic  Education  and  Communication  category: 

♦  Enable  users  to  document  and  communicate  descriptions  of  objectives,  measures,  targets,  and 
initiatives 

7.3.3  Business  Execution 

The  BSCol  suggests  the  following  features  for  the  Business  Execution  category: 

♦  Explicitly  show  the  relationship  between  initiatives  required  to  achieve  strategy 

7.3.4  Feedback  and  Learning 

The  BSCol  suggests  the  following  features  for  the  Feedback  and  Learning  category: 

♦  Measure  results  against  targets 

♦  Rely  on  objective  and  subjective  judgments,  for  example,  do  not  overrule  the  judgment  of  the  senior 
executive 

♦  Have  graphical  indicators  of  performance 

7.4  Available  Features 

During  the  software  review,  important  features  were  identified  by  the  research  team.  These  features 
ranged  from  network  compatibility  to  user  ease  and  friendliness.  The  presence  of  these  features  in  each  of 
the  three  software  packages  was  assessed  through  review  of  manufacturer  specifications.  The 
effectiveness  of  these  features  was  not  tested.  The  most  prominent  features  including  networking, 

interface,  aa3oo3mont  oapabilitioo,  and  optiona  are  dioouoood  next. 
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7.4.1  Networking  Features 


♦  Client/Server  Support  -  Software  package  can  be  used  over  an  intranet. 

♦  Cross  Platform  Support  -  Software  package  has  the  capability  to  use  additional  operating  systems 
other  than  Windows®,  such  as  UNIX®  or  Mac®. 

♦  Integration  capabilities  w/  existing  software  -  Software  package  can  use  data  from  existing  data 
collection  software  such  as  Online  Transaction  Processing  (OLTP)  or  Enterprise  Resource  Planning 
(ERP)  systems,  data  warehouse  systems,  and  others.  This  feature  is  very  important  to  quick 
integration  of  the  BSC. 

♦  Real-Time  Update  of  Information  -  Information  is  updated  instantly  instead  of  at  the  end  of  a  day  or 
week,  for  example 

♦  Web  support  -  A  web-supported  package  that  can  be  accessed  via  an  Internet  connection. 

7.4.2  Interface 

♦  Graphical  User  Interface  (GUI)  -  Software  package  uses  a  Windows®  style  point-and-click  interface. 

♦  Graphs  and  Trends  -  Graphs  and  trends  can  be  used  to  describe  measures. 

♦  Notes  for  Measures  and  Initiatives  -  Software  package  has  the  option  to  enter  notes,  descriptions, 
reasons  for  actions,  or  any  other  pertinent  information  that  assists  in  understanding  measures  or 
strategic  initiatives.  These  appear  with  the  measures  on  the  BSC. 

♦  Personal  Scorecard  -  Each  person  involved  in  the  BSC  can  have  a  personal  scorecard  that  focuses 
only  on  the  measures  over  which  they  have  ownership. 

♦  Report-Making  Wizard  -  Software  package  includes  programs  to  help  take  the  information  from  the 
software  package  and  create  hardcopy  reports. 

♦  Scorecard  Construction  Wizard  -  Software  package  includes  programs  or  macros  that  help  users 
construct  their  own  scorecards  without  extensive  knowledge  of  the  BSC. 

♦  Status  Warning  for  Measures  -  Software  package  sends  warnings  whenever  a  measure  leaves 
acceptable  parameters.  Examples  of  warnings  are  color-coded  indicators  and  e-mails  to  the  owner  of 
the  measure. 

♦  Strategic  Themes  -  Strategy  is  depicted  on-screen  within  the  software  package  to  easily  observe  how 
measures  and  initiatives  apply  to  it. 

♦  Wizard  to  Install  New  Measures  -  Software  package  helps  the  users  to  implement  new  measures 

without  vendor  assistance  or  extensive  training.  Packages  with  this  feature  are  more  rohust  and  can 

adapt  easily. 
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7.4.3  Assessment  Capabilities 

♦  Drill-Down  Capability  -  Features  let  a  higher-level  scorecard  break  down  its  aggregate  measures  into 
lower-level  measures.  This  is  useful  in  locating  problem  measures  that  are  outside  their  acceptable 
parameters. 

♦  Milestone  Markers  -  Milestone  markers  show  the  progress  toward  achieving  long-term  goals. 

♦  Quantitative  Analysis  -  Software  package  has  the  capability  to  perform  statistical  analysis  on 
information  received  from  the  measures,  such  as  averages  and  trends,  for  example. 

♦  Show  Cross-Linkage  of  Measures  -  Software  package  shows  the  interrelations  that  exist  between 
measures. 

7.4.4  Options 

♦  Access  Control  According  to  User  -  This  restricts  users  from  accessing  measures  outside  their 
responsibility  or  measures  that  are  higher  up  the  scorecards. 

♦  E-mail  Integration  -  There  is  an  integrated  e-mail  system  to  deliver  messages  about  measures,  notes, 
or  other  BSC-related  information. 

♦  Excel  Capabilities  -  Measures  can  be  integrated  into  Microsoft®  Excel  spreadsheets  for  further 
analysis. 

♦  Security  Features  -  These  are  features  such  as  user  login,  network  encryption,  and  other  features  that 
protect  the  data  in  the  scorecard. 

7.5  Selected  BSC  Software  Package  Comparison 

As  previously  mentioned,  three  BSC  software  packages,  ActiveStrategy  Enterprise™,  SPlmpact,  and 
pbviews,  w'ere  examined  and  compared.  A  brief  description  of  each  software  package  is  provided  in  this 
section.  In  addition,  the  matrix  in  Table  7.1  is  presented  to  summarize  the  features  of  each  software 
package.  The  criteria  used  to  evaluate  the  software  packages  include  the  BSCol  standards,  the  features 
described  in  Section  7.4,  and  team  judgment.  This  matrix  shows  that  pbviews  contains  all  the  preferred 
features. 

7.5.1  ActiveStrategy  Enterprise™ 

ActiveStrategy  releases  several  versions  of  their  BSC  softw'are  package.  These  range  from  a  basic 
scorecard  edition  to  their  Enterprise  edition  with  full  functionality  of  all  features  For  these  comparisons 
the  Enterprise  edition  was  examined. 
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ActiveStrategy  Enterprise™  focuses  on  measure  ownership  with  personal  scorecards  to  help  each  person 
keep  track  of  his  or  her  own  measures.  It  also  implements  security  features  so  that  each  person  only  has 
access  to  information  relevant  to  his  or  her  job.  It  features  drill-dow'n  capability'  to  help  divide  aggregate 
measures  for  analysis.  In  order  to  facilitate  organization  and  focus,  it  sends  relevant  reminders  and 
warnings  to  measure  owners. 

ActiveStrategy  Enterprise™  claims  to  be  a  “ready  out  of  the  box”  package.  ActiveStrategy  states  it  can  be 
fully  implemented  on  the  most  sophisticated  systems  within  eight  to  12  weeks.  Several  integration 
packages  are  included  to  help  ensure  total  compatibility.  The  system  can  work  over  an  intranet  and/or 
Internet.  Contact  information  for  ActiveStrategy  is  provided  in  Appendix  7. 

7.5.2  SPImpact 

Open  Ratings  claims  their  SPImpact  BSC  software  package  completely  follows  the  traditional  BSC 
methodology.  It  supports  all  basic  BSC  features,  including  drill-down  capability,  milestone  markers, 
multiple  scorecards,  and  quantitative  analysis.  It  also  supports  prioritizing  of  initiatives,  performance 
tracking,  and  display  of  interdependencies  among  units. 

SPImpact  employs  integration  software  for  most  data  sources.  It  also  integrates  with  company  e-mail 
systems  to  send  assessments,  reminders,  and  alerts.  SPImpact  includes  development  wizards  for 
scorecards  and  automated  system  maintenance  and  updating. 

SPImpact  has  an  out-of-the-box  set-up,  which  helps  reduce  initial  set-up  time.  The  software  package 
includes  security  features  such  as  enrollment  and  user  privileges.  This  allows  for  the  creation  and 
maintenance  of  multiple  scorecards.  Contact  information  for  Open  Ratings  is  provided  in  Appendix  7. 

7.5.3  pbviews 

Panorama  Business  Views  has  a  performance  measurement  software  package  called  pbviews.  pbviews 
supports  a  broad  range  of  performance  initiatives  including  the  BSC.  Their  software  package  is 
compatible  with  most  measurement  systems  and  existing  software.  For  those  that  are  not  directly 
compatible,  pbviews  employs  a  translating  system. 


The  roooaroh  team  took  part  in  a  presentation  by  a  Panorama  Business  Views  sales  representative. 

pbviews  has  a  user-friendly  interface.  Each  BSC  measure  has  a  range  of  descriptions  including  ownership 
(the  person  in  charge  of  the  measure),  notes,  strategic  goals,  and  the  ability  to  look  back  over  time  to  find 
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trends,  pbviews  supports  the  drill-down  function  (the  ability  to  go  down  through  each  measure  to  see  its 
•aggregates).  It  was  not  apparent  whether  linkages  could  be  formed  between  different  measures  and 
perspectives. 

The  software  package  is  deployable  over  an  intranet,  Internet,  or  both.  The  administrator  has  the  option  of 
restricting  access  to  measures  according  to  the  ownership  of  the  measure,  pbviews  allows  direct  input  of 
results  by  authorized  users.  Contact  information  for  Panorama  Business  Views  is  provided  in  Appendix  7. 


Table  7.1:  BSC  Software  Comparison  Matrix 


Features 

BSC  Software  Packages 

Network 

ActiveStrategy 

Enterprise™ 

SPImpact 

pbviews 

Client/Server  support 

X 

X 

X 

Cross-platform  support 

X 

X 

X 

Integration  capabilities  w/existing  software 

X 

X 

X 

Real-time  update  of  information 

X 

X 

X 

Web  support 

X 

X 

X 

Interface 

Graphical  user  interface 

X 

X 

X 

Graphs  and  trends 

X 

X 

X 

Notes  for  measures  and  initiatives 

X 

X 

X 

Personal  scorecard 

X 

X 

Report-making  wizard 

X 

X 

X 

Scorecard  construction  wizard 

X 

X 

X 

Status  warnings  for  measures 

X 

X 

X 

Strategic  themes 

X 

X 

X 

Wizard  to  Install  new  measuies 

X. 

X 

_ 
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-  Features 

BSC  Software  Packages 

Network 

ActiveStrategy 

Enterprise™ 

SPImpact 

pbviews 

Assessment  Capabilities 

Drill-down  capability 

X 

X 

X 

Milestone  markers 

X 

X 

X 

Quantitative  analysis 

X 

X 

X 

Show  cross-linkage  of  measures 

X 

X 

X 

Options 

Access  control  according  to  user 

X 

X 

X 

E-mail  integration 

X 

X 

X 

Microsoft®  Excel  capabilities 

X 

X 

Security  features 

X 

X 

X 
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Appendix  1:  Flightline  Maintenance  Process 

(Excerpted  from  Flightline  Orientation  Briefing,  10  March  2003) 


Aircraft  Landing 

The  beginning  of  the  maintenance  process  starts  when  an  aircraft  returns  from  a  mission.  The  table  below 
describes  the  events  involved. 


Event 

What  Happens 

Aircraft  returns  from  a  mission 

Before  the  aircraft  lands,  the  aircrew  may  relay  aircraft  or  system 
discrepancies  by  radio  to  the  flightline.  The  aircrew  generally  converts 
these  discrepancies  into  a  series  of  standard  quick-reference  codes  to 
expedite  transmission  and  to  allow  better  understanding  of  the  nature 
of  the  discrepancy. 

Final  landing  preparations 

Maintenance  personnel  begin  to  prepare  for  the  aircraft  landing. 
Maintenance  specialists  are  identified  to  work  reported  in-flight 
discrepancies. 

After  landing 

The  aircraft  taxies  to  or  near  its  designated  parking  location.  For  some 
aircraft,  the  aircrew  may  refuel  at  a  fuel  pit  before  taxiing  to  the 
assigned  parking  location. 

Parking 

Once  at  the  parking  location,  the  aircrew  may  power  down  systems 
and  engines.  In  the  event  of  an  Integrated  Combat  Turn  (ICT),  the 
aircrew  may  keep  certain  systems  and  engines  running  as 
maintenance  personnel  prepare  the  aircraft  for  an  immediate  launch. 

A  complete  discussion  of  ICTs  and  Quick  Turns  is  found  later  in  this 
chapter. 

Aircrew  departure 

■ 

. 

Once  the  aircraft  systems  and  engines  are  off,  the  aircrew  departs  the 
aircraft.  In  some  cases,  the  aircrew  will  discuss  discrepancies  with  the 
maintenance  specialist  at  the  aircraft.  Doing  so  provides  the 
maintainer  more  detailed  information  as  to  the  nature  of  the 
discrepancy  than  would  otherwise  be  available.  The  aircrew  should 
record  all  noted  in-flight  discrepancies  in  the  Aircraft  Forms  before 
departing  for  aircrew  debriefing.  An  aircraft  forms  binder  is  normally 
taken  to  aircrew  debriefing.  Maintenance  personnel  begin  their  work 
once  the  forms  are  returned  to  the  aircraft. 
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Parking  and  Recovery 

During  this  phase,  the  aircraft  is  “safed”  for  ground  operations  and  parked  at  its  designated  location. 
Though  the  sequence  of  events  may  vary  for  different  types  of  aircraft,  the  basic  process  is  the  same: 

♦  Landing  gear  pins  are  immediately  installed  to  prevent  the  gear  from  inadvertent  collapse; 

♦  Grounding  wires  are  installed,  and  aircraft  systems,  gun  systems,  and  munitions  are  “safed”; 

♦  Engine  oil  samples  are  taken  for  spectrometric  examination; 

♦  Aircraft  circuit  breakers  are  set,  streamers  placed,  and  protective  covering  positioned; 

♦  If  necessary,  the  aircraft  is  towed  to  its  designated  parking  location,  where  servicing  actions  begin 

Aircraft  Servicing 

Aircraft  servicing  actions  typically  include  checking  system  fluid  levels  and  lubrication.  The  most 
common  fluids  requiring  servicing  include  engine  oil,  hydraulic  fluid,  and  fuel.  The  next  scheduled 
mission  generally  dictates  fuel  loads;  however,  if  this  information  is  unavailable,  the  fuel  tanks  are  filled 
to  a  designated  minimum  “Ramp”  load.  Safety  is  paramount  during  any  fueling/de-fuel ing  operation. 
While  servicing  actions  take  place,  the  aircrew  is  debriefed. 

Aircrew  Maintenance  Debriefing 

An  aggressive  aircrew  maintenance  debriefing  program  is  essential  to  the  maintenance  process  to  ensure 
accurate  reporting  and  documentation  of  aircraft  malfunctions.  The  debriefing  session  is  the  cornerstone 
for  the  entire  maintenance  documentation  process;  it  documents  historical  data  for  identification  of  repeat 
and  recurring  discrepancies  to  assist  in  troubleshooting  and  correcting  malfunctions.  Documentation 
systems  used  by  the  debriefing  section  include  Core  Automated  Maintenance  System  (CAMS),  G081 
(CAMS  for  Airlift),  Tactical  Interim  CAMS/REMIS  Reporting  System  (T1CARRS),  or  Computerized 
Fault  Reporting  System  (CFRS). 

Debriefing  is  normally  conducted  at  the  end  of  each  sortie  or  abort.  Maintainers  use  debriefing  to 
document  in-flight  discrepancies,  aborts,  in-flight  emergencies,  flying  time  information,  Event  History 
Recorder  readings,  Operational  Check  Flights,  Functional  Check  Flights,  and  munitions  drops. 

The  debriefer  should  properly  identify  and  document  repeat  or  recurring  discrepancies,  as  well  as  landing 
status,  system  capability,  deviation,  and  system  fault  codes. 
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Repeat/Recurring  Discrepancies 

During  the  debriefing  process,  in-flight  discrepancies  are  reviewed  to  ensure  they  are  identified  as  a 
repeat  or  a  recurrence  when  necessary.  A  repeat  discrepancy  is  one  that  occurs  on  the  next  attempted 
sortie  after  corrective  action  is  complete.  Generally,  a  recurring  discrepancy  is  one  that  is  complete, 
although  this  may  differ  by  command. 

Land  Status  Codes 

Aircrew  and  maintainers  use  landing  status  codes  to  indicate  aircraft  status  upon  landing.  The  table  below 
lists  these  codes. 


Code 

Description 

1 

Aircraft  mission  capable  with  no  additional  discrepancies 

2 

Aircraft  or  system  has  minor  discrepancies  but  is  capable  of  further  mission  assignment  within 
normal  turnaround  times 

3 

Aircraft  or  system  has  major  discrepancies  in  mission-essential  equipment  that  may  require 
extensive  repair  to  replacement  prior  to  further  mission  assignment.  The  discrepancy  may  not 
affect  safety-of-flight  and  the  aircraft  may  be  Not  Mission  Capable  (NMC)  flyable 

8 

Aircraft  or  system  has  suspected  or  known  radiological/biological  contamination 

System  Capability  Codes 

These  codes  indicate  system  or  subsystem  capability  at  the  end  of  a  sortie.  The  table  below  describes 
these  codes. 


Code 

Description 

0 

System  flown  with  a  known  discrepancy,  no  additional  discrepancies  noted,  system  can  be 
used 

1 

System  used  and  performed  satisfactorily,  no  maintenance  required 

2 

System  used  and  performed  satisfactorily.  A  minor  malfunction  exists,  but  system  is  capable 
of  further  mission  assignment 

3 

System  performance  was  unsatisfactory,  this  system  did  not  cause  an  abort 

A 

System  porformanco  «/ae  unsatisfactory,  this  system  caused  nr  rnntrihi  ited  tn  an  ahnrt 

5 

System  out  of  commission  prior  to  takeoff 
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Description 


Code 


6 

System  installed  but  not  used 

7 

System  not  installed 

8 

Aircraft  or  system  has  suspected  or  known  radiological/biological  contamination 

Deviation  Codes 

The  codes  in  the  table  below  indicate  the  type  of  deviation  that  affect  the  flying  schedule. 


Code 


Description 


Air  Abort 


Air  Abort/ln-Flight  Emergency 


Cancellation 


Delay 


Early  Landing 


Early  Takeoff 


In-Flight  Incident 


Ground  Abort 


Late  Landing 


Late  Takeoff 


Subsequent  Late 


Spare 


Subsequent  Cancellation 


Tail  Number  Swap 


Operation  Deviation  Cause  Codes 

The  codes  in  the  table  below  indicate  the  reason  for  a  deviation  or  the  agency  that  caused  a  deviation  to 
the  flying  schedule. 

Note:  The  first  two  positions  are  constant,  but  the  complete  codes  vary  among  major  commands 
(MAJCOMs).  The  “x”  denotes  any  character  for  local  use. 


ATx 

Air  Traffic 

COx 

Contractor 

HQx 

Higher  Headquarters 

MTx 

Maintenance 

OPx 

Operations 

OTx 

Other 

SFx 

Material,  Safety  of  Flight 

SYx 

Sympathy 

SUx 

Supply 

TRx 

Tanker/Receiver  Deviation 

WXx 

Weather 

XXX 

Local  Option 

System  Fault  Codes 

System  fault  codes  exist  for  aircraft  using  fault  reporting  and  fault  isolation  manuals.  These  codes  help 
maintainers  gain  access  to  correct  fault  isolation  procedures.  The  codes  vary  depending  on  the 
malfunction.  The  CFRS,  if  available,  uses  software  that  automates  the  identification  of  system  fault  codes. 

Post-Flight  Inspection 

After  flight,  an  aircraft  will  undergo  a  Thru-Flight  Inspection,  a  Basic  Post-Flight  Inspection,  or  a 
Combined  Pre-Flight/  Basic  Post-Flight  or  Pre-Flight/Thru-Flight  Inspection.  The  following  table 
describes  some  of  these  inspections. 
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Type  Inspection 

Description 

Thru-Flight 

The  Thru-Flight  Inspection  is  a  between-flights  inspection  accomplished  after 
each  flight  when  a  turn-around  sortie  or  continuation  flight  is  scheduled  and  a 

Basic  Post-Flight  Inspection  is  not  required.  This  inspection  consists  of  checking 
the  aircraft  for  flight  continuance  by  performing  visual  examination  or  operational 
checks  of  certain  components,  areas,  or  systems  to  assure  that  no  defects  exist, 
which  would  be  detrimental  to  further  flight. 

Post-flight 

The  Basic  Post-Flight  inspection  occurs  after  the  last  flight  of  a  specified  flying 
period.  This  inspection  will  consist  of  checking  the  aircraft  condition  by  performing 
visual  examination  or  operational  checks  of  certain  components,  areas,  or 
systems  to  assure  that  no  defects  exist  that  would  be  detrimental  to  flight.  It  is  a 
more  thorough  check  that  the  Pre-Flight  or  the  Thru-Flight  Inspections. 

Combined 

The  Combined  Pre-Flight/Basic  Post-Flight  Inspection  or  Pre-Flight/Thru-Flight 
Inspections  consolidates  the  requirements  of  the  Pre-Flight  and  Basic  Post-Flight 
Inspections  into  a  single  inspection  accomplished  at  the  end  of  the  flying  period.  It 
eliminates  duplication  of  inspection  items  and  is  valuable  during  periods  of  high 
aircraft  generation  rates. 

Unscheduled  Maintenance/Repair 

During  this  phase  of  the  process,  maintenance  technicians  start  work  on  the  in-flight  discrepancies 
reported  by  the  aircrew.  The  figure  below  and  the  accompanying  table  illustrate  and  explain  the  process 
for  completing  unscheduled  maintenance  actions. 


72 


73 


Action 

Description 

Identify 

discrepancy 

Once  a  discrepancy  is  identified,  the  aircraft  crew  chief  requests 

♦  troubleshooting 

♦  ordering  parts  (if  needed) 

♦  determining  whether  munitions  can  be  “safed”  or  downloaded 

♦  adjustment/repair 

♦  operational  checks 

♦  aircraft  forms  and  Maintenance  Management  Information  System  (MMIS) 
documentation 

Work 

discrepancy 

Once  the  technician  troubleshoots  the  discrepancy,  he  or  she  faces  three  alternatives: 

♦  If  the  technician  cannot  duplicate  the  discrepancy  and  has  exhausted  all  possible 
courses  of  action,  he  or  she  clears  the  discrepancy  by  indicating  “cannot 
duplicate”  (CND)  in  the  aircraft  forms  and  the  MMIS.  CND  actions  are  a  last  resort 
and  are  coordinated  through  the  Expediter  and  Production  Superintendent. 

♦  The  system  or.subsystem  may  require  adjustments  or  repair  action  that  does  not 
require  a  replacement  part.  Once  corrective  actions  and  operational  checks  are 
complete,  the  technician  clears  the  discrepancy  by  indicating  the  corrective 
actions  in  the  aircraft. 

♦  The  technician  may  need  to  change  a  system  or  subsystem  part  or  Line 

Replaceable  Unit  (LRU).  Once  the  technician  identifies  which  part  requires 
replacement,  the  technician  orders  it. 

If 

Then 

The  part  is  available  in  base 
supply 

It  is  issued  to  the  technician  and  installed  on  the 
aircraft 

The  part  is  not  available 

♦  The  part  is  backordered 

♦  The  Production  Superintendent  is  notified  of  the 
backorder  status  and  asked  to  establish  an  order 
priority 

♦  The  repair  effort  for  the  discrepancy  is  placed  on 
hold  until  the  required  part  arrives  through  the 
base 

♦  In  certain  cases,  cannibalization  procedures  may 
supply  the  part 

When  the  part 
is  available 

♦  It  is  issued  to  the  technician  and  installed  on  the  aircraft 

♦  Operational  checks  are  performed  on  the  system  and  subsystem 

♦  The  job  is  completed  and  signed  off  in  aircraft  forms 
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Preventive  Maintenance 

Preventive  maintenance  includes  all  scheduled  maintenance  actions  performed  to  retain  the  aircraft  and  its 
systems  in  mission-ready  condition.  Scheduled  maintenance  involves  the  accomplishment  of  periodic 
inspections,  condition  monitoring,  Time  Compliance  Technical  Orders  (TCTO),  Time  Change  Item  (TCI) 
replacements,  and  system  calibrations.  The  maintenance  schedule  controls  preventive  maintenance 
activities.  Close  coordination  between  the  production  superintendent,  flight  chiefs,  aircraft  schedulers, 
and  the  support  flight  ensures  all  necessary  resources  are  available  to  perform  the  job.  These  resources 
include  the  aircraft,  manpower,  parts,  and  equipment. 

The  Expediter  or  Production  Superintendent  notifies  the  aircraft  crew  chief  and  maintenance  technicians 
of  any  scheduled  maintenance  activities  either  during  shift  roll  call,  or  by  having  them  review  the  daily 
maintenance  schedule.  The  crew  chief  or  maintenance  technician  is  responsible  for  obtaining  the 
necessary  tools,  equipment,  and  parts  from  their  support  flight  and,  if  necessary,  supply.  Once  the  job  is 
complete,  they  should  notify  the  expediter  and  clear  (sign-off)  the  write  up  in  the  aircraft  forms  and 
MMIS  by  indicating  the  maintenance  actions  taken. 

Aircraft  Scheduled  for  Next  Mission 

While  the  aircraft  is  undergoing  corrective  and  preventive  maintenance,  it  may  be  scheduled  to  fly  its  next 
mission.  The  flying  requirement  for  a  particular  aircraft  is  documented  in  the  monthly  and  weekly  flying 
schedules.  The  production  superintendent  closely  monitors  these  schedules  to  ensure  the  aircraft  are  ready 
for  the  next  scheduled  flying  mission.  As  this  time  approaches,  the  maintainers  prepare  the  aircraft  for  its 
mission  and  for  flight. 

Aircraft  Mission  Preparation 

During  this  phase,  many  activities  may  occur  simultaneously.  Final  corrective  and  preventive 
maintenance  actions  are  completed;  fuel  adjustments  are  made  to  accommodate  last-minute  mission 
changes;  munitions,  chaff,  ammunition,  specialized  pods,  and  specialized  equipment  are  loaded  and 
configured.  The  aircraft  crew  chief  is  responsible  for  ensuring  the  aircraft  is  ready  for  its  mission.  All 
these  activities  must  be  closely  monitored  and  coordinated  to  ensure  nothing  is  overlooked. 

Pre-Launch  Inspection 

Pre-launch  activities  involve  detailed  maintenance  and  aircrew  inspections.  Brief  explanations  of  the  Pre- 
Flight  and  aircrew  inspections  follow.  TO  00-20-5  contains  a  complete  discussion  of  aircraft  inspections. 
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Pre-Flight  Inspection 

A  Pre-Flight  Inspection  is  a  flight  preparedness  check  that  the  crew  chief  does  in  accordance  with  the 
aircraft.  This  inspection  includes  visually  examining  the  aircraft  and  operationally  checking  certain 
systems  and  components  to  ensure  there  are  no  serious  defects  or  malfunctions. 

Aircrew  Inspection 

The  aircrew  accomplishes  the  aircrew  inspection,  commonly  referred  to  as  the  “Dash  one  inspection,”  in 
accordance  with  the  aircraft  aircrew  manual.  This  inspection  includes  a  visual  examination  of  the  aircraft 
and  may  require  the  aircrew  to  configure  certain  systems  in  preparation  for  launch  and  mission. 

Once  the  inspections  are  complete,  the  aircrew  and  maintainers  carefully  review  aircraft  forms  to  ensure 
all  discrepancies  are  cleared.  An  exceptional  release  is  required.  This  is  the  final  check  performed  by 
maintenance  before  they  release  the  aircraft  to  the  aircrew. 

Aircraft  Launch 

At  this  point,  the  aircrew  starts  the  engines,  powers  up  systems  and  makes  final  adjustments  in 
preparation  for  launch.  Maintainers  disconnect  and  move  the  support  equipment  away  from  the  aircraft. 
When  cleared  by  the  control  tower,  the  crew  chief  marshals  the  aircraft  out  of  its  parking  spot  and  onto 
the  ramp  or  taxi  way.  Some  aircraft  require  an  End-of-Runway  Inspection.  The  purpose  of  this  inspection 
is  to  detect  critical  defects  that  may  have  developed  or  have  become  apparent  during  ground  operation  of 
the  aircraft.  It  is  performed  immediately  prior  to  take-off  at  a  designated  location  usually  near  the  end  of 
the  runway.  Some  units  publish  a  local  checklist  for  end-of-runway  procedures. 

Post-Launch  Cleanup 

Finally,  after  the  aircraft  is  airborne,  the  maintainers  clean  up  the  parking  location.  Personnel  must  store 
all  items  such  as  the  fire  extinguisher,  trash  receptacle,  tire  chock,  and  inlet  covers  in  their  proper 
locations.  A  Foreign  Object  Damage  (FOD)  check  should  also  occur  to  ensure  someone  removes  objects 
that  may  have  blown  onto  the  parking  spot  during  aircraft  taxi. 
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Appendix  2:  Exhaustive  List  of  Performance  Measures 


Exhaustive  List  of  Performance  Measures 


Performance  Measure 

9302 

12-Hour  Fix  Rate 

Yes 

12-Hour  Fixes  Completed 

Yes 

4-Hour  Fix  Rate 

Yes 

4-Hour  Fixes  Completed 

Yes 

8-Hour  Fix  Rate 

Yes 

8-Hour  Fixes  Completed 

Yes 

Measure  Sources 


HOF 

Reports 


Adherence  to  OPS/MXS  Schedule 


8  Air  Abort  Rate 


9  Air  Aborts 


1 0  Average  Repair  Time  (Fighters) 


1 1  Average  Repair  Time  (Other) 


12  Average  Time  as  Hangar  Queen 


13 

Backup  Aircraft  Inventory  (BAI) 

14 

Break  Rate 

15 

Cannibalization  Rate 

16 

Career  Development  Course  Success  Rate 

17 

Cat  1  Hangar  Queens 

18 

Cat  II  Hangar  Queens 

19 

Cat  III  Hangar  Queens 

20 

Chargeable  Deviation© 

21 

Code  3  Breaks 
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Exhaustive  List  of  Performance  Measures 


Performance  Measure 


22  Could  Not  Duplicate  Rate 


23  Deferred  Discrepancies  Leading  to  RR  Instances 


24  Deferred  Discrepancies  Leading  to  TNMC  or  PMC 


25  Deferred  Discrepancy  Rate  AWM 


26  Deferred  Discrepancy  Rate  AWP 


27  Departure  Reliability 


28  Depot  Scheduling 


29  Discrepancies  Awaiting  Maintenance 


30  Discrepancies  Awaiting  Parts 


31  EEO  Complaints 


32  Electronic  Warfare  Pod  Mission-Capable  Rate 


33  Functional  Check  Flight  (FCF)  Release  Rate 


34  Fix  Rate 


35  Flying  Schedule  Effectiveness 


36  Fully  Mission-Capable  Rate 


37  Ground  Aborts 


38  Hourly  Utilization  Rate  (UTE) 


39  Hours  Flown 


40  Hours  Programmed 


41  IG  Complaints 


42  Issue  Effectiveness  Rate 


43  Job  Data  Documentation  Error  Rate 


44  Judicial  Punishments 


Measure  Sources 

9302 

HOF 

Reports 

Metrics 

Manual 

Yes 

Yes 

Yes 

Yes 

Ye 

s 
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Exhaustive  List  of  Performance  Measures 


# 

Performance  Measure 

45 

Maintenance  Man-Hours  per  Flying  Hour 

46 

Maintenance  Scheduling  Effectiveness 

47 

Mean  Time  to  Failure 

48 

Mean  Time  to  Repair 

49 

Mission-Capable  Rate 

50 

Mission-Impaired  Capability  Awaiting  Parts 

51 

Non-Chargeable  Deviations 

52 

Non-Judicial  Punishments 

53 

Primary  Aircraft  Inventory  (PAI) 

54 

Partially  Mission-Capable  Maintenance 

55 

Partially  Mission-Capable  Supply 

56 

Phase  Average 

57 

Process  Improvements 

58  Promotion  Average 


59  Quarterly  Annual  Awards 


60  Repeat-Recur  Rate 


61  Scheduled  Hours 


62  Sortie  UTE 


63  Sorties  Flown 


64  Sorties  Programmed 


65  Sorties  Scheduled 


66  Spare  Aircraft  Inventory 


67  Special  Experience  Identifier 


Measure  Sources 

9302 

HOF 

Reports 

Metrics 

Manual 

Exhaustive  List  of  Performance  Measures 


Performance  Measure 


68  Stockage  Effectiveness  Rate 


69  Time  Compliance  Technical  Orders  Finished 


70  Total  Abort  Rate 


71  Total  Accumulated  Cycles 


72  Total  Aircraft  Control  Deviations 


73  Total  Deviations 


74  Total  Higher  Command  Deviations 


75  Total  Maintenance  Deviations 


76  Total  Operations  Deviation 


77  Total  Repair  Cycle  Time 


78  Total  Supply  Deviations 


79  Total  Weather  Deviations 


80  Totally  Not  Mission-Capable  Maintenance 


81  Totally  Not  Mission-Capable  Supply 


82  Training  No-Shows 


83  Training  Overdues 


84  Training  Schedule  Adherence 


85  Unit  Average  Skill  Level 


86  Upgrade  Training  Status 


87  Weather 


Measure  Sources 

9302 

HOF 

Reports 

Metrics 

Manual 
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Appendix  3:  Family  to  Measure  Association 


Family 


Family  to  Measure  Association 


Measure 


Excellence 


Air  Abort  Rate 


Air  Aborts 


Break  Rate 


Code  3  Breaks 


Could  Not  Duplicate  Rate 


FCF  Release  Rate 


Job  Data  Documentation  Error  Rate 


Mean  Time  to  Failure 


Mean  Time  to  Repair 
Repeat  Recur  Rate 


Total  Abort  Rate 


Total  Accumulated  Cycles 


Growth  Career  Development  Course  Success  Rate 
EEO  Complaints 
IG  Complaints 


Judicial  Punishments 


Non-Judicial  Punishments 


Process  Improvements 
Promotion  Average 
Quarterly  Annual  Awards 
Special  Experience  Identifier 
Training  No-Shows 


Training  Overdues 
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Family  to  Measure  Association 


Family 


Measure 


Training  Schedule  Adherence 


Unit  Average  Skill  Level 


Upgrade  Training  Status 


Average  Repair  Time  (Fighters) 


Average  Repair  Time  (Other) 


Backup  Aircraft  Inventory 


Electronic  Warfare  Pod  Mission-Capable  Rate 


Fully  Mission-Capable  Rate 


Hours  Flown 


Hours  Programmed 

Productivity  Maintenance  Man-Hours  per  Flying  Hour 
Mission-Capable  Rate 
Primary  Aircraft  Inventory  (PA1) 


Partially  Mission-Capable  Maintenance 


Sortie  UTE 


Sorties  Flown 


Sorties  Programmed 


Totally  Not  Mission-Capable  Maintenance 


Scheduling  4-Hour  Fix  Rate 


8-Hour  Fix  Rate 


12-Hour  Fix  Rate 


Chargeable  Deviations 


Depot  Scheduling 


Fix  Rate 


Supply 


Family  to  Measure  Association 


Measure 


Maintenance  Scheduling  Effectiveness 
Non-Chargeable  Deviations 
Phase  Average 
Scheduled  Hours 
Sorties  Scheduled 

Time  Compliance  Technical  Orders  Finished 
Total  Aircraft  Control  Deviations 
Total  Deviations 

Total  Higher  Command  Deviations 
Total  Operations  Deviations 
Total  Weather  Deviations 
Weather 

Average  Time  as  Hangar  Queen 

Cannibalization  Rate 

Cat  I  Hangar  Queens 

Cat  II  Hangar  Queens 

Cat  III  Hangar  Queens 

Discrepancies  Awaiting  Parts 

Issue  Effectiveness  Rate 


Mission-Impaired  Capability  Awaiting  Parts  Fill  Rate 
Partially  Mission-Capable  Supply 
Spare  Aircraft  Inventory 


Family  to  Measure  Association 


Family 

Measure 

Stockage  Effectiveness  Rate 

Total  Supply  Deviations 

Totally  Not  Mission-Capable  Supply 

Timeliness 

4-Hour  Fixes  Completed 

8-Hour  Fixes  Completed 

12-Hour  Fixes  Completed 

Adherence  to  OPS/MXS  Schedule 

Deferred  Discrepancies  Leading  to  RR  Instances 

Deferred  Discrepancies  Leading  to  TNMC  or  PMC 

Deferred  Discrepancy  Rate  AWM 

Deferred  Discrepancy  Rate  AWP 

Departure  Reliability 

Discrepancies  Awaiting  Maintenance 

Ground  Aborts 

Total  Maintenance  Deviations 

Total  Repair  Cycle  Time 
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Appendix  4:  BSC  Validation  Questionnaire 


UNIVERSITY  OF  ARKANSAS 


In  support  of  research  to  develop  a  “health-of-fleet”  type  metric,  please  rank  the  aircraft  maintenance 
performance  metrics  below  in  order  of  decreasing  criticality. 

1)  Rank  each  of  the  four  Perspectives  (categories)  in  order  of  decreasing  criticality  (1  =  most  critical) 

2)  Within  each  Perspective,  add  any  critical  metrics  that  we  omitted 

3)  Rank  all  metrics  within  each  Perspective  in  order  of  decreasing  criticality  (1  =  most  critical) 


Mission  Perspective  (Rank _ ) 

-  Long  term  ability  to  improve  combat  capability 

Maintenance  hours  per  flying  hour 
MC  rate 
PMCM 
Sorties  flown 
'  TNMCM 


Influencing  Factors  Perspective  (Rank  _ ) 

-  Success  with  collaborators  and  functional  processes 

CANN  rate 
’  MICAP  fill  rates 

Maintenance  Scheduling  Effectiveness 
’  TNMCS 


Management  Perspective  (Rank _ ) 

-  Achievement  of  customer  service 

Adherence  to  OPS/MXS  schedule 

DD  rate  AWM 

Total  maintenance  deviations 

4-hour  fix  rate 

8-hour  fix  rate 

12-hour  fix  rate 

Days  in  Phase/ISO 


Internal  Enhancement  Perspective  (Rank 
-  Necessity  for  innovation  and  growth 

_ CND  rate 

_ Repeat-Recur  (RR)  rate 

_ Special  Experience  Identifiers 

_ Total  abort  rate 

_ Training  schedule  adherence 

_ Upgrade  Training 

_ Unit  average  technical  skill  level 


Please  check  all  of  the  applicable  boxes  regarding  your  job  description  below  for  response  categorization: 


Group-level  mxs  supervision/staff 
Squadron-level  mxs  supervision/staff 
Flight-level  mxs  supervision/staff 
Production  Supervisor 
Expeditor 


Flight  Chief 
Lead  Technician 
Flightline  Maintainer 
Other 
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Appendix  5:  Questionnaire — Raw  Data  Spreadsheet 
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Upgrade  Training  j 
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Level  ! 


ISI 
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Job  Descrii 


Appendix  6:  BSCol  Functional  Standards  Brochure 
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1.0  Background 

Within  the  past  few  years*  the  Balanced  Scorecard  has  become  a  key  fool  for  managing  and  implementing  strategy  in 
organizations.  Responding  to  this  business  demand,  many  analytical  application,  KRP,  and  database  vendors  are 
developing  and  releasing  software  to  support  executive  decision-making  using  the  Balanced  Scorecard  methodology. 
This  development  of  software-enabled  Balanced  Scorecards  will  inevitably  lead  to  rhe  implementation  of  Balanced 
Scorecards  at  an  increasing  number  of  organizations  throughout  the  world.  Centra!  to  t  he  Balanced  Scorecard 
methodology  is  holistic  vision  of  a  measurement  system  tied  to  the  strategic  direction  of  the  firm..  The  Balanced 
Scorecard  is  based  on  strategic  implementation  through  focus  on  four  perspectives,  with  financial  objectives  and 
measures  supported  by  customer,  internal,  and  learning,  and  growth  objectives  and  metrics.  By  measuring  and 
managing  the  business  using  this  balanced  set  of  measures,  an  organization  can  ensure  rapid  and  effective 
implementation  of  strategy  and  facilitate  organizational  alignment  and  communication. 

Many  organizations  are  accelerating:  the  benefits  from  their  Balanced  Scorecard  by  using  enabling  technology  to 
facilitate  their  Balanced  Scorecard  management  process.  If  an  organization  is  to  capture  the  full  potential  of  the 
Balanced  Scorecard  management  system,  the  enabling  technology  it  implements  should  support  the  requirements  of 
the  desired  management  process.  If  the  elements  of  the  management  process  are  not  reflected  in  the  application,  the 
resulting  process  will  be  sub-optimal.  <  uirreiith ,  many  software  vendors  are  offering  applications  that  bear  the  name 
Balanced  Scorecard.  Some  of  these  applications  support  a  strategic  management  process,  while  others  do  not  .  This 
has  made  die  Balanced  Scorecard  technology  marketplace  confusing  for  potential  buyers. 

To  facilitate  consistent  and  appropriate  use  of  the  Balanced  Scorecard  globally,  a  need  has  arisen  for  harmonization 
and  standardization  of  the  methodology  of  the  Balanced  Scorecard  as  envisioned  by  tin  creators  of  the  concept.  1  )r. 
Robert  Kaplan  and  Dr.  David  Norton.  Therefore,  Balanced  Scorecard  Collaborative,  a  global  center  of  excellence  on 
all  things  related  to  Balanced  Scorecards,  founded  by  Drs.  Kaplan  and  Norton,  has  created  Balanced  Scorecard 
Functional  St  a  nd.mb. 

Balanced  Scorecard  Functional  Standards  identify  user  requirements  and  needs,  based  on  rhe  experience  professionals 
from  the  Balanced  Sawecard  Collaborative  have  bail  with  more  than  300  clients.  These  observations  have  been 
codified  in  the  functional  standards  to  pro\  idc  guidance  for  organizations  preparing  ti  >  purchase  a  Balanced  Scorecard 
application  and  to  provide  a  functional  baseline  for  technology  vendors  developing  a  Balanced  Scorecard  application. 
These  standards  should  be  viewed  as  a  minimum  threshold  upon  which  vendors  will  innovate  to.  meet  any  additional 
requirements  of  their  market. 

The  purpose  of  this  document  is  to  provide  an  overview  of  Balanced  Scorecard  Functional  Standards. 

2.0  Document  References 

The  functional  standards  listed  below  are  based  on  rhe  published  methodology  of  Drs.  Kaplan  and  Norton  and  the 
expertise  of  the  staff  of  Balanced  Scorecard  Collaborative.  For  more  information  about  the  methodology  behind  the 
standards,  and  flu-  best-in-class  application  of  Balanced  Scorecard  methodology,  ph  ase  refer  to  the  following 
documents: 

"Linking,  the  Balanced  Scorecard  to  Strategy"  in  Coi/forniit Nanjgfmit!  RovVw,  by  Robert  S.  Kaplan  and  David  P. 
Norton,  October  19%. 

’llvo  U.aVoioo,4  Oitftwnvrl  ►•m*'  4oe<vli  l<>  rtioneu|ti<  !•  r.tla  curroHr  op.o'nttoi.r  »v  •»!  «Ke  ilHc-vi'e  *  ■>  t 

future  performance.  Many  managers  believe  they  are  using  a  Balanced  Scorecard  when  they  supplement  traditional 
financial  measures  with  generic,  nun-financial  measures  about  customers,  processes,  and  employees.  But  the  best 
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Balanced  Storecards  arc  more  than  ad  hot  collections  of  financial  and  non-financial  measures.  The  objectives  anti 
measures  on  a  Balanced  Scorecard  should  he  derived  From  the  business  unit's  strategy.  A  scorecard  should  contain 
outcome  measures  and  the  performance  drivers  of  those  outcomes,  linked  together  in  cause-asul-cffect  relationships. 

Balanced  Scorecard:  Translating  Strategy  into  Action,  hv  Robert  S.  Kaplan  and  David  P.  Norton  (i  btrvard  Business 
School  Press,  19%). 

Abstract:  The  Balanced  Scorecard  translates  a  company’s  vision  anti  strategy  info  a  coherent  set  of  performance 
measures.  The  four  perspectives  of  the  scorecard — financial  measures,  customer  knowledge,  internal  business 
processes,  and  learning  and  growth— offer  a  balance  between  short-term  ami  long-term  objectives,  Ivrween  outcomes 
desired  and  performance  drivers  of  those  outcomes,  and  between  bard  objective  measures  and  softer,  more  subjective 
measures.  In  the  first  part,  Kaplan  and  Norton  provide  the  theoretical  foundations  for  the  Balanced  Scorecard;  in  the 
second  part,  they  describe  the  sups  organizations  must  take  to  build  their  own  Scorecards;  and,  finally,  thee  discuss 
lmw  the  Balanced  Scorecard  can  be  used  as  a  driver  of  change. 

"Using  the  Balanced  Scorecard  as  a  Strategic  Management  System"  to  Harvard  Ikwutf.*  KnvVxv,  by  Robert  S.  Kaplan  and 
David  P.  Norton,  limitary  19%. 


Abstract:  As  companies  transform  themselves  to  compete  in  the  world  of  information,  their  ability  to  exploit 
intangible  assets  is  becoming  more  decisive  than  their  ability  to  manage  physical  assets.  Several  years  ago,  Robert  S. 
Kaplan  and  David  P.  Norton  introduced  the  balanced  scorecard,  which  enabled  companies  to  track  financial 
results  while  monitoring  progress  in  building  the  capabilities  they  would  need  for  growth.  Recently,  some  companies 
have  gone  further  and  discovered  the  scorecard's  value  as  the  cornerstone  of  a  new  strategic  management  system. 
Traditional  management  systems  rely  on  financial  measures,  which  bear  little  relation  to  progress  in  achieving  long- 
tarn  strategic  objectives.  The  scorecard  introduces  font  new  processes  that  help  companies  connect  long-term 
objectives  with  short-term  actions. 


"Putting  the  Balanced  Scorecard  to  Work"  in  hianun-f  hmxeff  Rotes 'by  Robert  S.  Kaplan  and  David  P.  Norton, 
September  19%. 

Abstract:  In  an  earlier,  groundbreaking  article,  "Balanced  Scorecard  —  Measures  That  Drive  Performance1',  the  authors 
proposed  a  new  measurement  system  that  provided  managers  with  a  comprehensive  framework  to  translate  a 
company’s  strategic  objectives  into  a  coherent  set  of  performance  measures.  Now  the  authors  show  how  several 
Companies  tire  putting  the  balanced  scorecard  to  work,  liffective  measurement,  the  authors  point  out,  must  be  an 
integral  part  of  the  management  process.  Much  more  than  a  measurement  exercise,  the  balanced  scorecard  is  a 
management  system  that  can  motivate  breakthrough  improvements  in  such  critical  areas  as  product,  process, 
customer,  and  market  development.  Several  examples — Rockwatcr,  Apple  Computer,  and  Advanced  Micro 
Devices — illustrate  how  the  scorecard  combines  measurement  anil  management  in  different  companies.  Prom 
the  experiences  of  these  companies  and  others,  the  authors  have  found  that  the  balanced  scorecard  is  most  successful 
when  it  is  used  to  drive  the  process  of  change. 


"The  Balanced  Scorecard:  Measures  That  Drive  Performance"  in  Ihinur.i  llusine/f  Rti'U.'i  by  Robert  S.  Kaplati  and 
David  P.  Norton,  January  1992. 

Abstract:  During  a  year-long  research  project,  the  authors  dev  eloped  a  "balanced  scorecard"  performance 
measurement  system  that  allows  executives  to  view  a  company  from  several  perspectives  simultaneously.  'The 
scorecard  includes  financial  measures  that  reveal  the  results  of  actions  already  taken,  as  w  ell  as  three  sets  of 


,t 
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improve.  Creating  a  balanced  scorecard  requires  translating  a  company’s  strategy  and  mission  statement  into  specific 
goals  and  measures.  Managers  then  track  those  measures  as  they  work  toward  their  goals. 
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3.0  Definitions 

i 1  ( it  rite  purpose  of  this  document,  the  follow  ing  definitions  apply: 

Bafiilh'r/fSatiiWirtfh  a  multi-dimensional  framework  created  by  Dr.  Robert  Kaplan  and  Dr.  David  Norton  that  uses 
measurement  to  describe  an  organization's  strategy  (see  references  in  Section  2.0). 

Ba/dnnJ Saatiiiri  Applkaikn  refers  to  any  softw  are  package  which  uses  the  methodology  of  Drs.  Norton  and  Kaplan  to 
facilitate  strategic  decision-making  using  the  Balanced  Scorecard  nicrhodologv.  or  am-  package  which  uses  tin-  term 
’’Balanced  Scorecard"  in  its  marketing  material,  title,  or  external  communications. 

SktMhmh  as  defined  in  this  document  refer  only  to  the  functionality  of  the  software  package  as  it  relates  to  the 
Balanced  Scorecard  methodology.  This  standard  does  not  specify  am  technology  solutions,  refer  to  scalabiim  of  the 
solution,  or  discuss  interconnectivity  of  the  package  with  other  systems. 

UalMtX/i Smfa'ttrfi hltlhoihkg;  refers,  in  the  context  of  these  standards,  to  the  Balanced  Scorecard  methodology  as 
initially  defined  hv  Drs.  Kaplan  and  Norton  and  as  currently  practiced  by  Balanced  Scorecard  Collaborative.  The 
methodology  is  under  continual  development,  and  the  standards  w  ill  evoh  e  ro  reflect  current  best  practices  and 
thought  leadership  in  the  Balanced  Scorecard  concept. 

4.0  Balanced  Scorecard  Collaborative 

Balanced  Scorecard  Collaborative,  Inc.  facilitates  the  worldwide  aw  areness,  use,  enhancement,  ami  integrity  of  the 
Balanced  .Scorecard  as  a  value-added  management  process.  Tin-  O  dial*  native  offers  a  variety  of  education,  training, 
research,  and  development  services  designed  to  share  Ix-st  practices  to  achieve  best  results,  bounded  and  managed  by 
Balanced  Scorecard  creators  Drs.  Robert  Kaplan  and  David  Norton,  the  Collaborative  prov  ides  organizations  and 
indiv  iduals  w  ith  a  global  cent  er  of  excellence  anil  expertise  on  all  things  related  to  Balanced  Scorecards. 


Balanced  Scorecard  applications  must  he  able  to  facilitate  management  of  tlu-  organization  through  the  Balanced 
Scorecard  methodology  as  documented  in  the  literature  abov  e  and  as  practiced  by  Balanced  Scorecard  <  '.olt.ibor.uiu-. 
The  standards  as  outlined  below  represent  the  minimum  functionality  for  a  Balanced  Scorecard  system.  Software 
prov  iders  are  encouraged  to  differentiate  their  products  be;  ond  the  minimum  standards  outlined  below. 

'flic  standards  documentation  has  been  divided  into  four  sections: 

The  application  should  be  able  to  flexibly  accommodate  the  basic  elements  of  a  proper  Balanced  Scorecard  design. 
The  application  must  be  able  to  (!)  view  the  strategy  from  four  perspectives  (financial,  customer,  internal,  ami 
learning),  f2)  identify  strategic  objectiv  es  for  each  perspective,  (3;  associate  measures  w  ith  strategic  objectiv  es,  (4)  link 
strategic  objectives  in  cause  and  effect  relationships,  {5»  assign  targets  ro  measures,  and  Hi)  list  strategic*  initiatives. 

Strategic  Kducarion  and  Communication  'Section  5.2: 

One  of  the  key  reasons  for  implementing  a  Balanced  Scorecard  software  solution  is  the  facilitation  of  strategic 
education  and  communication.  Therefore,  a  certified  application  will  enable  users  to  document  and  communicate 
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Business  Hxfcuritnt  'Section  5.3; 

Initiatives  (discretionary  investment  programs)  are  the  testing grounds'  for  the  strategy  expressed  in  the  Balanced 
Scorecard.  Therefore,  a  certified  application  insist  make  explicit  the  relationship  between  initiatives  required  to  achieve 
the  strategy  and  rhe  associated  strategic  objective. 

Feedback  and  I  a-aroimr  (Section  5.4> 

Through  proper  system  design,  rhe  feedback  cycle  time  for  management  information  can  be  significantly  reduced. 
Analysis  of  the  measure  result  s  against  targets  will  allow  managers  to  understand  which  areas  of  the  organization 
require  further  attention,  f  Imvever,  the  system  should  not  override  the  judgment  of  a  sensor  executive  —  the  Balanced 
Scorecard  system  should  rely  on  both  objective  and  subjective  judgments,  as  well  as  graphical  indicators,  to 
report  on  progress  of  a  particular  measure  against  its  target.  These  design  principles  are  outlined  in  greater  detail  in 
the  text  that  follows. 

As  of  September  30.1999,  Balanced  Scorecard  Collaborative  considers  these  the  minimum  functional  standards  that  a 
Balanced  Scorecard  application  should  have  in  order  to  reflect  the  methodology  endorsed  by  Balanced  Scorecard 
Collaborative.  Release  1.0a,  published  on  May  5, 2000  includes  minor  modifications  that  d*>  nor  affect  the  substance 
of  the  functional  standards. 

5.1  Balanced  Scorecard  Design 

A  certified  Balanced  Scorecard  application  will  accommodate  the  basic  elements  of  a  proper  Balanced  Scorecard 
design.  Naming  conventions  may  differ,  but  rhe  design  structure  of  certified  applications  must  include  rhe  following 
six  features:  (see  Figure  1,  Basic  Balanced  Scorecard  Design,  for  an  example) 

1.  Perspective* 

A  perspective  is  a  component  into  which. the  strategy  is  decomposed  n>  drive  implementation,  typically,  there  are 
four  perspectives;  financial,  customer,  internal  and  learning  and  growth.  I  fibers  may  be  added  or  replace  these  based 
on  a  specific  strategic  need.  A  perspective  is  a  major  element  of  the  strategy  often  representing  a  stakeholder  category 
or  point  of  view.  Certified  applications  will  include  at  least  four  basic  perspectives  (financial,  customer,  internal 
processes,  learning  and  growth)  and  have  the  ability  to  rename  perspectives  at  the  user’s  option. 

.2.  Objectives 

An  objective  is  a  statement  of  strategic  intent.  An  objective  states  how  a  strategy  will  lx*  made  operational.  <  Aneralh, 
the  objectives  form  the  building  blocks  for  rite  overall  strategy  of  die  organization.  Certified  applications  will  allow 
strategic  objectives  to  be  aligned  with  at  least  one  perspectiv  e. 

1  Measures 

A  measure  is  a  performance  metric  that  will  reflect  progress  against  an  object h  e.  A  measure  must  he  quantifiable,  'live 
measures  communicate  the  specific  Ixiuu  ior  required  to  achieve  the  objective  and  become  rhe  actionable  statement 
of  how  the  strategic  objective  will  be  accomplished.  Leading  measures  ate  predictors  of  future  performance,  while 
lagging  measures  are  outcomes.  A  certified  application  will  allow  a  reasonable  number  of  measures  explicitly  linked  to 
at  least  one  objective. 

i.  lUlfcOi 

A  target  is  a  quantifiable  goal  for  each  measure.  The  set  of  targets  found  on  the  Balanced  Scorecard  become  the 
overall  goals  of  the  organization.  Targets  create  opportunity  to  succeed,  help  the  organization  monitor  progress 

it m  .till  AMBUT'iL  jj’iiitfo,  auU  unimiuuK.iu  cAjnA.i.mluu».  A  kviiifltU  a*  ill  dllwn  vjuatnilTaUK-  iiiili  *i 
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Objectives  an-  related  to  one  another  through  cause  and  effect  relationships.  The  cause  ami  effect  linkages  are  similar 
to  "if-then”  statements,  for  example,  if  an  airline  decreases  the  on-ground  turn-around  lime  {objective  1 5,  then  the 
airline  will  require  fewer  planes  (objective  2)  (//;// customers  will  be  more  satisfied  with  on-time  take  off  (objective  pi 
and  corporate  profitability  will  increase  (objective  4).  'Hu  so  cause  and  effect  linkages  should  be  explicit.  A  certified 
application  will  allow  objectives  to  be  linked  and  graphically  represented  on-screen  as  a  series  of  cause  and  effect 
linkages  {strategy  map).  The  linkages  should  be  able  to  be  easily  changed  and  edited  as  apprt>priaie. 

6.  Strategic  Iciti  it iv  es 

Strategic  initiatives  are  those  action  programs  (discretionary  inv  vs  t  incuts  or  projects)  that  drive  strategic  performance. 
These  are  the  activ  ilies  that  groups  will  focus  on  to  ensure  attainment  ol  strategic  result*.  All  initiatives  undcrw ay  in  an 
organization  should  be  aligned  with  the  strategy  in  the  Balanced  Scorecard.  A  compliant  package  will  allow  for  a  set 
of  strategic  initiatives  to  be  linked  to  at  least  one  objective. 

5.2  Strategic  Education  and  Communication 

Certified  applications  must  include  the  following  functionality: 


The  six  key  elements  of  the  Balanced  Scorecard  (perspectiv  es.  objectives,  measures,  targets,  linkages  and  iiuti.mv  i  s; 
are  typically  defined  in  greater  levels  of  detail,  for  instance,  objuliivs  may  lie  further  defined  through  a  sentence  or  two 
describing  the  strategic  situation  or  issue.  -Mtasttm  mid  are  iv  pica  IK  broken  down  into  formulas,  units  of 

measure,  frequency  of  reporting,  target  owner,  reporting,  responsibility,  data  sources,  target  effective  date,  and  target 
history  .  Ixi/iii/i’vs  also  require  greater  description  such  as  timeline,  resources,  budget,  benefits.  and  risks.  Such 
documentation  is  important  to  users  to  ensure  consistency  and  repeatability  ol'  reporting  and  to  capture  the  strategic 
intent  of  each  element.  Certified  applications  will  facilitate  the  documentation  of  qualitative  descriptions  of  each 
element  of  the  Balanced  Scorecard. 


5.3  Business  Execution 


Strategic  initiatives  are  die  base  elements  of  work  that  drive  strategic  change,  they  are  the  discretionary  programs 
such  as  training,  adv  ertising,  reengineering  and  others  that  an-  put  in  place  to  make  the  strategy  happen.  They  must  be 
monitored  on  a  continuous  basts  to  insure  that  they  are  being  implemented  as  planned  and  producing  the  desired 
results.  A  strategic  initiativ  e  should  be  explicitly  linked  or  mapped  to  achiev  ing  one  or  more  strategic  objective. 
Eerlified  applications  should  be  able  to  display  matching  initiatives  and  objectives  ansi  should 
allow  initiatives  to  be  tied  to  more  than  one  strategic  objectiv  es.  (Reference  f  igure  1  for  example: 

5.4  Feedback  and  Learning 

BaadLml  Ikpumttf ; 

Base  level  reporting  includes  reporting  of  pcrformanci  data  for  each  measure.  I  listorical  performance  against  targets 
and  multiple  reporting  views  are  desirable  features.  Terrified  applications  should  be  able  to  display  current 
performance  data  for  each  measure. 
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subjectiv  ely  analyze  the  data.  The  subjective  performance  assessments  should  also  be  backed  up  by  u  memo-style 
qualitative  assessment  of  die  .external  or  internal  variables  underlying  the  assessment.  Certified  applications  will 


BALANCED 

SCORECARD 

CO!  I  »ROitaiivr 

/ 


Hji.ns.rd  S;vff amt  i  :m 


*'ihl  .'j  lif&iti  f&i ' 

ki’A  tjht 


permit  subjective  assessments  of  performance  (c.g.,  veil,  yellow,  erven)  as  well  as  memo-style  ijnalirain  <•  descriptions 
of  performance. 

N'isnal  Status  Indicators 

Hitch  element  of  the  scorecard  must  have  a  visual  indicator  of  performance  status,  c.g.,  green  or  a  plus  sign  if  a 
measure  or  objective  is  on  plan,  red  or  a  minus  sign  if  behind  plan,  {terrified  applications  will  graphically  display 
performance  against  targets  in  an  easv-to-comprehcnd  format,  ami  will  allow  for  modification  to  suit  individual  end- 
user  needs. 

6.0  Conclusion 

It  is  anticipated  that:  bv  following  these  guidelines  soft's  are  developers  will  lie  able  to  develop  Balanced  Scorecard 
applications,  which  enable  users  ro  capture  the  benefit  of  the  Balanced  Scorecard  muri.igenu-ni  svstem.  Buyers  of 
Balanced  Scorecard  applications  will  also  hem-fir,  as  they  will  he  able  to  leverage  the  expertise  of  Balanced  Scorecard 
Collaborative  and  numerous  successful  organizations,  as  expressed  it)  these  standards,  to  select  appropriate  Balanced 
Scorecard  software  that  will  meet  their  needs. 

Figure  1:  Basic  Scorecard  Design  (Example) 
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Appendix  7:  Contact  Information  for  Software  Packages 

Active  Strategy 
Michael  Brazukas 
Director  of  Marketing 
Active  Strategy,  Inc. 

(610) 239-8517 

brazukas@activestrategy.com 

www.activestrategy.com/software/enterprise.html 

Open  Ratings 

Bruce  Thomson 

Authorized  Agent  for  Open  Ratings,  Inc. 

5  Star  Partners 

(847)612-0030 

brucet@5starpartners.com 

www.openratings.com/capabilities/spimpact/ 

Panorama  Business  Views 
David  L.  Parks 
Sales  Manager 

Panorama  Business  Views  (USA) 

(888)  241-4201  or  (416)  525-3700  (cell) 
dparks@pbviews.com 

www.pbviews.com/products/features/features.asp 
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